SherbyNet - /page/wikibook/entry.mu

<!DOCTYPE html>

<title>Multi-task learning</title>

</head>

<span id="openzim-page-title" class="mw-page-title-main"><span class="mw-page-title-main">Multi-task learning</span></span>

</h1>

</header>

<p><b>Multi-task learning</b> (MTL) is a subfield of <a href="Machine_learning" title="Machine learning">machine learning</a> in which multiple learning tasks are solved at the same time, while exploiting commonalities and differences across tasks. This can result in improved learning efficiency and prediction accuracy for the task-specific models, when compared to training the models separately.<sup id="cite_ref-1" class="reference"><a href="#cite_note-1"><span class="cite-bracket">[</span>1<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-2" class="reference"><a href="#cite_note-2"><span class="cite-bracket">[</span>2<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-:2_3-0" class="reference"><a href="#cite_note-:2-3"><span class="cite-bracket">[</span>3<span class="cite-bracket">]</span></a></sup>

Inherently, Multi-task learning is a <a href="Multi-objective_optimization" title="Multi-objective optimization">multi-objective optimization</a> problem having <a href="Trade-off" title="Trade-off">trade-offs</a> between different tasks.<sup id="cite_ref-4" class="reference"><a href="#cite_note-4"><span class="cite-bracket">[</span>4<span class="cite-bracket">]</span></a></sup>

Early versions of MTL were called "hints".<sup id="cite_ref-5" class="reference"><a href="#cite_note-5"><span class="cite-bracket">[</span>5<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-6" class="reference"><a href="#cite_note-6"><span class="cite-bracket">[</span>6<span class="cite-bracket">]</span></a></sup>

</p><p>

In a widely cited 1997 paper, Rich Caruana gave the following characterization:</p><blockquote><p>Multitask Learning is an approach to <a href="Inductive_transfer" class="mw-redirect" title="Inductive transfer">inductive transfer</a> that improves <a href="Generalization_error" title="Generalization error">generalization</a> by using the domain information contained in the training signals of related tasks as an <a href="Inductive_bias" title="Inductive bias">inductive bias</a>. It does this by learning tasks in parallel while using a shared <a href="Representation_learning" class="mw-redirect" title="Representation learning">representation</a>; what is learned for each task can help other tasks be learned better.<sup id="cite_ref-:2_3-1" class="reference"><a href="#cite_note-:2-3"><span class="cite-bracket">[</span>3<span class="cite-bracket">]</span></a></sup></p></blockquote>

<p>In the classification context, MTL aims to improve the performance of multiple classification tasks by learning them jointly. One example is a spam-filter, which can be treated as distinct but related classification tasks across different users. To make this more concrete, consider that different people have different distributions of features which distinguish spam emails from legitimate ones, for example an English speaker may find that all emails in Russian are spam, not so for Russian speakers. Yet there is a definite commonality in this classification task across users, for example one common feature might be text related to money transfer. Solving each user's spam classification problem jointly via MTL can let the solutions inform each other and improve performance. Further examples of settings for MTL include <a href="Multiclass_classification" title="Multiclass classification">multiclass classification</a> and <a href="Multi-label_classification" title="Multi-label classification">multi-label classification</a>.<sup id="cite_ref-:1_7-0" class="reference"><a href="#cite_note-:1-7"><span class="cite-bracket">[</span>7<span class="cite-bracket">]</span></a></sup>

</p><p>Multi-task learning works because <a href="Regularization_(mathematics)" title="Regularization (mathematics)">regularization</a> induced by requiring an algorithm to perform well on a related task can be superior to regularization that prevents <a href="Overfitting" title="Overfitting">overfitting</a> by penalizing all complexity uniformly. One situation where MTL may be particularly helpful is if the tasks share significant commonalities and are generally slightly under sampled.<sup id="cite_ref-:bmdl_8-0" class="reference"><a href="#cite_note-:bmdl-8"><span class="cite-bracket">[</span>8<span class="cite-bracket">]</span></a></sup> However, as discussed below, MTL has also been shown to be beneficial for learning unrelated tasks.<sup id="cite_ref-:bmdl_8-1" class="reference"><a href="#cite_note-:bmdl-8"><span class="cite-bracket">[</span>8<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-:3_9-0" class="reference"><a href="#cite_note-:3-9"><span class="cite-bracket">[</span>9<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading2"><h2 id="Methods">Methods</h2></div>

<p>The key challenge in multi-task learning, is how to combine learning signals from multiple tasks into a single model. This may strongly depend on how well different task agree with each other, or contradict each other. There are several ways to address this challenge:

</p>

<div class="mw-heading mw-heading3"><h3 id="Task_grouping_and_overlap">Task grouping and overlap</h3></div>

<p>Within the MTL paradigm, information can be shared across some or all of the tasks. Depending on the structure of task relatedness, one may want to share information selectively across the tasks. For example, tasks may be grouped or exist in a hierarchy, or be related according to some general metric. Suppose, as developed more formally below, that the parameter vector modeling each task is a <a href="Linear_combination" title="Linear combination">linear combination</a> of some underlying basis. Similarity in terms of this basis can indicate the relatedness of the tasks. For example, with <a href="Sparse_array" class="mw-redirect" title="Sparse array">sparsity</a>, overlap of nonzero coefficients across tasks indicates commonality. A task grouping then corresponds to those tasks lying in a subspace generated by some subset of basis elements, where tasks in different groups may be disjoint or overlap arbitrarily in terms of their bases.<sup id="cite_ref-10" class="reference"><a href="#cite_note-10"><span class="cite-bracket">[</span>10<span class="cite-bracket">]</span></a></sup> Task relatedness can be imposed a priori or learned from the data.<sup id="cite_ref-:1_7-1" class="reference"><a href="#cite_note-:1-7"><span class="cite-bracket">[</span>7<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-11" class="reference"><a href="#cite_note-11"><span class="cite-bracket">[</span>11<span class="cite-bracket">]</span></a></sup> Hierarchical task relatedness can also be exploited implicitly without assuming a priori knowledge or learning relations explicitly.<sup id="cite_ref-:bmdl_8-2" class="reference"><a href="#cite_note-:bmdl-8"><span class="cite-bracket">[</span>8<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-12" class="reference"><a href="#cite_note-12"><span class="cite-bracket">[</span>12<span class="cite-bracket">]</span></a></sup> For example, the explicit learning of sample relevance across tasks can be done to guarantee the effectiveness of joint learning across multiple domains.<sup id="cite_ref-:bmdl_8-3" class="reference"><a href="#cite_note-:bmdl-8"><span class="cite-bracket">[</span>8<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading3"><h3 id="Exploiting_unrelated_tasks">Exploiting unrelated tasks</h3></div>

<p>One can attempt learning a group of principal tasks using a group of auxiliary tasks, unrelated to the principal ones. In many applications, joint learning of unrelated tasks which use the same input data can be beneficial. The reason is that prior knowledge about task relatedness can lead to sparser and more informative representations for each task grouping, essentially by screening out idiosyncrasies of the data distribution. Novel methods which builds on a prior multitask methodology by favoring a shared low-dimensional representation within each task grouping have been proposed. The programmer can impose a penalty on tasks from different groups which encourages the two representations to be <a href="Orthogonal" class="mw-redirect" title="Orthogonal">orthogonal</a>. Experiments on synthetic and real data have indicated that incorporating unrelated tasks can result in significant improvements over standard multi-task learning methods.<sup id="cite_ref-:3_9-1" class="reference"><a href="#cite_note-:3-9"><span class="cite-bracket">[</span>9<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading3"><h3 id="Transfer_of_knowledge">Transfer of knowledge</h3></div>

<p>Related to multi-task learning is the concept of knowledge transfer. Whereas traditional multi-task learning implies that a shared representation is developed concurrently across tasks, transfer of knowledge implies a sequentially shared representation. Large scale machine learning projects such as the deep <a href="Convolutional_neural_network" title="Convolutional neural network">convolutional neural network</a> <a href="GoogLeNet" class="mw-redirect" title="GoogLeNet">GoogLeNet</a>,<sup id="cite_ref-13" class="reference"><a href="#cite_note-13"><span class="cite-bracket">[</span>13<span class="cite-bracket">]</span></a></sup> an image-based object classifier, can develop robust representations which may be useful to further algorithms learning related tasks. For example, the pre-trained model can be used as a feature extractor to perform pre-processing for another learning algorithm. Or the pre-trained model can be used to initialize a model with similar architecture which is then fine-tuned to learn a different classification task.<sup id="cite_ref-14" class="reference"><a href="#cite_note-14"><span class="cite-bracket">[</span>14<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading3"><h3 id="Multiple_non-stationary_tasks">Multiple non-stationary tasks</h3></div>

<p>Traditionally Multi-task learning and transfer of knowledge are applied to stationary learning settings. Their extension to non-stationary environments is termed <i>Group online adaptive learning</i> (GOAL).<sup id="cite_ref-15" class="reference"><a href="#cite_note-15"><span class="cite-bracket">[</span>15<span class="cite-bracket">]</span></a></sup> Sharing information could be particularly useful if learners operate in continuously changing environments, because a learner could benefit from previous experience of another learner to quickly adapt to their new environment. Such group-adaptive learning has numerous applications, from predicting <a href="Financial_modeling" title="Financial modeling">financial time-series</a>, through content recommendation systems, to visual understanding for adaptive autonomous agents.

</p>

<div class="mw-heading mw-heading3"><h3 id="Multi-task_optimization">Multi-task optimization</h3></div>

<p><b>Multi-task optimization</b> focuses on solving optimizing the whole process.<sup id="cite_ref-TO_16-0" class="reference"><a href="#cite_note-TO-16"><span class="cite-bracket">[</span>16<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-mfo_17-0" class="reference"><a href="#cite_note-mfo-17"><span class="cite-bracket">[</span>17<span class="cite-bracket">]</span></a></sup> The paradigm has been inspired by the well-established concepts of <a href="Transfer_learning" title="Transfer learning">transfer learning</a><sup id="cite_ref-18" class="reference"><a href="#cite_note-18"><span class="cite-bracket">[</span>18<span class="cite-bracket">]</span></a></sup> and multi-task learning in <a href="Predictive_analytics" title="Predictive analytics">predictive analytics</a>.<sup id="cite_ref-19" class="reference"><a href="#cite_note-19"><span class="cite-bracket">[</span>19<span class="cite-bracket">]</span></a></sup>

</p><p>The key motivation behind multi-task optimization is that if optimization tasks are related to each other in terms of their optimal solutions or the general characteristics of their function landscapes,<sup id="cite_ref-20" class="reference"><a href="#cite_note-20"><span class="cite-bracket">[</span>20<span class="cite-bracket">]</span></a></sup> the search progress can be transferred to substantially accelerate the search on the other.

</p><p>The success of the paradigm is not necessarily limited to one-way knowledge transfers from simpler to more complex tasks. In practice an attempt is to intentionally solve a more difficult task that may unintentionally solve several smaller problems.<sup id="cite_ref-DeFreitas_21-0" class="reference"><a href="#cite_note-DeFreitas-21"><span class="cite-bracket">[</span>21<span class="cite-bracket">]</span></a></sup>

</p><p>There is a direct relationship between multitask optimization and <a href="Multi-objective_optimization" title="Multi-objective optimization">multi-objective optimization</a>.<sup id="cite_ref-22" class="reference"><a href="#cite_note-22"><span class="cite-bracket">[</span>22<span class="cite-bracket">]</span></a></sup>

</p><p>In some cases, the simultaneous training of seemingly related tasks may hinder performance compared to single-task models.<sup id="cite_ref-23" class="reference"><a href="#cite_note-23"><span class="cite-bracket">[</span>23<span class="cite-bracket">]</span></a></sup> Commonly, MTL models employ task-specific modules on top of a joint feature representation obtained using a shared module. Since this joint representation must capture useful features across all tasks, MTL may hinder individual task performance if the different tasks seek conflicting representation, i.e., the gradients of different tasks point to opposing directions or differ significantly in magnitude. This phenomenon is commonly referred to as negative transfer. To mitigate this issue, various MTL optimization methods have been proposed. Commonly, the per-task gradients are combined into a joint update direction through various aggregation algorithms or heuristics.

</p><p>There are several common approaches for multi-task optimization: <a href="Bayesian_optimization" title="Bayesian optimization">Bayesian optimization</a>, <a href="Evolutionary_computation" title="Evolutionary computation">evolutionary computation</a>, and approaches based on <a href="Game_theory" title="Game theory">Game theory</a>.<sup id="cite_ref-TO_16-1" class="reference"><a href="#cite_note-TO-16"><span class="cite-bracket">[</span>16<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading4"><h4 id="Multi-task_Bayesian_optimization">Multi-task Bayesian optimization</h4></div>

<p><b>Multi-task Bayesian optimization</b> is a modern model-based approach that leverages the concept of knowledge transfer to speed up the automatic <a href="Hyperparameter_optimization" title="Hyperparameter optimization">hyperparameter optimization</a> process of machine learning algorithms.<sup id="cite_ref-mtbo_24-0" class="reference"><a href="#cite_note-mtbo-24"><span class="cite-bracket">[</span>24<span class="cite-bracket">]</span></a></sup> The method builds a multi-task <a href="Gaussian_process" title="Gaussian process">Gaussian process</a> model on the data originating from different searches progressing in tandem.<sup id="cite_ref-25" class="reference"><a href="#cite_note-25"><span class="cite-bracket">[</span>25<span class="cite-bracket">]</span></a></sup> The captured inter-task dependencies are thereafter utilized to better inform the subsequent sampling of candidate solutions in respective search spaces.

</p>

<div class="mw-heading mw-heading4"><h4 id="Evolutionary_multi-tasking">Evolutionary multi-tasking</h4></div>

<p><b>Evolutionary multi-tasking</b> has been explored as a means of exploiting the <a href="Implicit_parallelism" title="Implicit parallelism">implicit parallelism</a> of population-based search algorithms to simultaneously progress multiple distinct optimization tasks. By mapping all tasks to a unified search space, the evolving population of candidate solutions can harness the hidden relationships between them through continuous genetic transfer. This is induced when solutions associated with different tasks crossover.<sup id="cite_ref-mfo_17-1" class="reference"><a href="#cite_note-mfo-17"><span class="cite-bracket">[</span>17<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-cognitive_26-0" class="reference"><a href="#cite_note-cognitive-26"><span class="cite-bracket">[</span>26<span class="cite-bracket">]</span></a></sup> Recently, modes of knowledge transfer that are different from direct solution <a href="Crossover_(genetic_algorithm)" class="mw-redirect" title="Crossover (genetic algorithm)">crossover</a> have been explored.<sup id="cite_ref-27" class="reference"><a href="#cite_note-27"><span class="cite-bracket">[</span>27<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-28" class="reference"><a href="#cite_note-28"><span class="cite-bracket">[</span>28<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading4"><h4 id="Game-theoretic_optimization">Game-theoretic optimization</h4></div>

<p>Game-theoretic approaches to multi-task optimization propose to view the optimization problem as a game, where each task is a player. All players compete through the reward matrix of the game, and try to reach a solution that satisfies all players (all tasks). This view provide insight about how to build efficient algorithms based on <a href="Gradient_descent" title="Gradient descent">gradient descent</a> optimization (GD), which is particularly important for training <a href="Deep_neural_networks" class="mw-redirect" title="Deep neural networks">deep neural networks</a>.<sup id="cite_ref-29" class="reference"><a href="#cite_note-29"><span class="cite-bracket">[</span>29<span class="cite-bracket">]</span></a></sup> In GD for MTL, the problem is that each task provides its own loss, and it is not clear how to combine all losses and create a single unified gradient, leading to several different aggregation strategies.<sup id="cite_ref-30" class="reference"><a href="#cite_note-30"><span class="cite-bracket">[</span>30<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-31" class="reference"><a href="#cite_note-31"><span class="cite-bracket">[</span>31<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-32" class="reference"><a href="#cite_note-32"><span class="cite-bracket">[</span>32<span class="cite-bracket">]</span></a></sup> This aggregation problem can be solved by defining a game matrix where the reward of each player is the agreement of its own gradient with the common gradient, and then setting the common gradient to be the Nash <a href="Cooperative_bargaining" title="Cooperative bargaining">Cooperative bargaining</a><sup id="cite_ref-33" class="reference"><a href="#cite_note-33"><span class="cite-bracket">[</span>33<span class="cite-bracket">]</span></a></sup> of that system.

</p>

<div class="mw-heading mw-heading2"><h2 id="Applications">Applications</h2></div>

<p>Algorithms for multi-task optimization span a wide array of real-world applications. Recent studies highlight the potential for speed-ups in the optimization of engineering design parameters by conducting related designs jointly in a multi-task manner.<sup id="cite_ref-cognitive_26-1" class="reference"><a href="#cite_note-cognitive-26"><span class="cite-bracket">[</span>26<span class="cite-bracket">]</span></a></sup> In <a href="Machine_learning" title="Machine learning">machine learning</a>, the transfer of optimized features across related data sets can enhance the efficiency of the training process as well as improve the generalization capability of learned models.<sup id="cite_ref-34" class="reference"><a href="#cite_note-34"><span class="cite-bracket">[</span>34<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-35" class="reference"><a href="#cite_note-35"><span class="cite-bracket">[</span>35<span class="cite-bracket">]</span></a></sup> In addition, the concept of multi-tasking has led to advances in automatic <a href="Hyperparameter_optimization" title="Hyperparameter optimization">hyperparameter optimization</a> of machine learning models and <a href="Ensemble_learning" title="Ensemble learning">ensemble learning</a>.<sup id="cite_ref-36" class="reference"><a href="#cite_note-36"><span class="cite-bracket">[</span>36<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-37" class="reference"><a href="#cite_note-37"><span class="cite-bracket">[</span>37<span class="cite-bracket">]</span></a></sup>

</p><p>Applications have also been reported in cloud computing,<sup id="cite_ref-38" class="reference"><a href="#cite_note-38"><span class="cite-bracket">[</span>38<span class="cite-bracket">]</span></a></sup> with future developments geared towards cloud-based on-demand optimization services that can cater to multiple customers simultaneously.<sup id="cite_ref-mfo_17-2" class="reference"><a href="#cite_note-mfo-17"><span class="cite-bracket">[</span>17<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-39" class="reference"><a href="#cite_note-39"><span class="cite-bracket">[</span>39<span class="cite-bracket">]</span></a></sup> Recent work has additionally shown applications in chemistry.<sup id="cite_ref-40" class="reference"><a href="#cite_note-40"><span class="cite-bracket">[</span>40<span class="cite-bracket">]</span></a></sup> In addition, some recent works have applied multi-task optimization algorithms in industrial manufacturing.<sup id="cite_ref-41" class="reference"><a href="#cite_note-41"><span class="cite-bracket">[</span>41<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-42" class="reference"><a href="#cite_note-42"><span class="cite-bracket">[</span>42<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading2"><h2 id="Mathematics">Mathematics</h2></div>

<div class="mw-heading mw-heading3"><h3 id="Reproducing_Hilbert_space_of_vector_valued_functions_(RKHSvv)">Reproducing Hilbert space of vector valued functions (RKHSvv)</h3></div>

<p>The MTL problem can be cast within the context of RKHSvv (a <a href="Complete_metric_space" title="Complete metric space">complete</a> <a href="Inner_product_space" title="Inner product space">inner product space</a> of <a href="Vector-valued_function" title="Vector-valued function">vector-valued functions</a> equipped with a <a href="Reproducing_kernel_Hilbert_space" title="Reproducing kernel Hilbert space">reproducing kernel</a>). In particular, recent focus has been on cases where task structure can be identified via a separable kernel, described below. The presentation here derives from Ciliberto et al., 2015.<sup id="cite_ref-:1_7-2" class="reference"><a href="#cite_note-:1-7"><span class="cite-bracket">[</span>7<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading4"><h4 id="RKHSvv_concepts">RKHSvv concepts</h4></div>

<p>Suppose the training data set is <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {S}}_{t}=\{(x_{i}^{t},y_{i}^{t})\}_{i=1}^{n_{t}}}">

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {S}}_{t}=\{(x_{i}^{t},y_{i}^{t})\}_{i=1}^{n_{t}}}</annotation>

</semantics>

</math></span><img src="./b3648e3008d345892be61ba8b4ee526d32090503.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:17.544ex; height:3.176ex;" alt="{\displaystyle {\mathcal {S}}_{t}=\{(x_{i}^{t},y_{i}^{t})\}_{i=1}^{n_{t}}}" loading="lazy"></span>, with <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle x_{i}^{t}\in {\mathcal {X}}}">

</mrow>

</mrow>

</msubsup>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle x_{i}^{t}\in {\mathcal {X}}}</annotation>

</semantics>

</math></span><img src="./e9addb6c4e70f514fe168c853d1ddf62ce4abbd9.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:6.872ex; height:3.176ex;" alt="{\displaystyle x_{i}^{t}\in {\mathcal {X}}}" loading="lazy"></span>, <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle y_{i}^{t}\in {\mathcal {Y}}}">

</mrow>

</mrow>

</msubsup>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle y_{i}^{t}\in {\mathcal {Y}}}</annotation>

</semantics>

</math></span><img src="./6cd3c672204ffbfba06152f753875f860b4d4c51.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:6.487ex; height:3.176ex;" alt="{\displaystyle y_{i}^{t}\in {\mathcal {Y}}}" loading="lazy"></span>, where <span class="texhtml mvar" style="font-style:italic;">t</span> indexes task, and <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle t\in 1,...,T}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle t\in 1,...,T}</annotation>

</semantics>

</math></span><img src="./86105b9314831a91aa24f014d2a9aeb26a8d5a27.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:11.649ex; height:2.509ex;" alt="{\displaystyle t\in 1,...,T}" loading="lazy"></span>. Let <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle n=\sum _{t=1}^{T}n_{t}}">

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle n=\sum _{t=1}^{T}n_{t}}</annotation>

</semantics>

</math></span><img src="./8a8b0fbe98ac4eb2188f4380685dd3948356a43b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:10.456ex; height:7.343ex;" alt="{\displaystyle n=\sum _{t=1}^{T}n_{t}}" loading="lazy"></span>. In this setting there is a consistent input and output space and the same <a href="Loss_function" title="Loss function">loss function</a> <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {L}}:\mathbb {R} \times \mathbb {R} \rightarrow \mathbb {R} _{+}}">

</mrow>

</mrow>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {L}}:\mathbb {R} \times \mathbb {R} \rightarrow \mathbb {R} _{+}}</annotation>

</semantics>

</math></span><img src="./c0610604544e461cbfb685f601ab77f1a25bde89.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:16.54ex; height:2.509ex;" alt="{\displaystyle {\mathcal {L}}:\mathbb {R} \times \mathbb {R} \rightarrow \mathbb {R} _{+}}" loading="lazy"></span> for each task: . This results in the regularized machine learning problem:

</p>

/* start https://en.wikipedia.org/ */

.mw-parser-output table.numblk{border-collapse:collapse;border:none;margin-top:0;margin-right:0;margin-bottom:0}.mw-parser-output table.numblk>tbody>tr>td{vertical-align:middle;padding:0}.mw-parser-output table.numblk>tbody>tr>td:nth-child(2){width:99%}.mw-parser-output table.numblk>tbody>tr>td:nth-child(2)>table{border-collapse:collapse;margin:0;border:none;width:100%}.mw-parser-output table.numblk>tbody>tr>td:nth-child(2)>table>tbody>tr:first-child>td:first-child,.mw-parser-output table.numblk>tbody>tr>td:nth-child(2)>table>tbody>tr:first-child>td:last-child{padding:0 0.4ex}.mw-parser-output table.numblk>tbody>tr>td:nth-child(2)>table>tbody>tr:first-child>td:nth-child(2){width:100%;padding:0}.mw-parser-output table.numblk>tbody>tr>td:nth-child(2)>table>tbody>tr:last-child>td{padding:0}.mw-parser-output table.numblk>tbody>tr>td:last-child{font-weight:bold}.mw-parser-output table.numblk.numblk-raw-n>tbody>tr>td:last-child{font-weight:unset}.mw-parser-output table.numblk>tbody>tr>td:last-child::before{content:"("}.mw-parser-output table.numblk>tbody>tr>td:last-child::after{content:")"}.mw-parser-output table.numblk.numblk-raw-n>tbody>tr>td:last-child::before,.mw-parser-output table.numblk.numblk-raw-n>tbody>tr>td:last-child::after{content:none}.mw-parser-output table.numblk>tbody>tr>td{border:none}.mw-parser-output table.numblk.numblk-border>tbody>tr>td{border:thin solid}.mw-parser-output table.numblk>tbody>tr>td:nth-child(2)>table>tbody>tr:first-child>td{border:none}.mw-parser-output table.numblk.numblk-border>tbody>tr>td:nth-child(2)>table>tbody>tr:first-child>td{border:thin solid}.mw-parser-output table.numblk>tbody>tr>td:nth-child(2)>table>tbody>tr:last-child>td{border-left:none;border-right:none;border-bottom:none}.mw-parser-output table.numblk.numblk-border>tbody>tr>td:nth-child(2)>table>tbody>tr:last-child>td{border-left:thin solid;border-right:thin solid;border-bottom:thin solid}.mw-parser-output table.numblk:target{color:var(--color-base,#202122);background-color:#cfe8fd}@media screen{html.skin-theme-clientpref-night .mw-parser-output table.numblk:target{color:var(--color-base,#eaecf0);background-color:#301702}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output table.numblk:target{color:var(--color-base,#eaecf0);background-color:#301702}}

/* end https://en.wikipedia.org/ */

</style><table role="presentation" class="numblk" style="margin-left: 1.6em;"><tbody><tr><td class="nowrap"><span class="mwe-math-element mwe-math-element-block" id="1"><span class="mwe-math-mathml-display mwe-math-mathml-a11y" style="display: none;"><math display="block" xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \min _{f\in {\mathcal {H}}}\sum _{t=1}^{T}{\frac {1}{n_{t}}}\sum _{i=1}^{n_{t}}{\mathcal {L}}(y_{i}^{t},f_{t}(x_{i}^{t}))+\lambda ||f||_{\mathcal {H}}^{2}}">

</mrow>

</munder>

</mrow>

</mrow>

</munderover>

<mfrac>

<msub>

</mrow>

</msub>

</mfrac>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</munderover>

</mrow>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{f\in {\mathcal {H}}}\sum _{t=1}^{T}{\frac {1}{n_{t}}}\sum _{i=1}^{n_{t}}{\mathcal {L}}(y_{i}^{t},f_{t}(x_{i}^{t}))+\lambda ||f||_{\mathcal {H}}^{2}}</annotation>

</semantics>

</math></span></span></td> <td></td> <td class="nowrap"><span id="math_1" class="reference nourlexpansion" style="font-weight:bold;">1</span></td></tr></tbody></table>

<p>where <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {H}}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {H}}}</annotation>

</semantics>

</math></span><img src="./19ef4c7b923a5125ac91aa491838a95ee15b804f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.964ex; height:2.176ex;" alt="{\displaystyle {\mathcal {H}}}" loading="lazy"></span> is a vector valued reproducing kernel Hilbert space with functions <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f:{\mathcal {X}}\rightarrow {\mathcal {Y}}^{T}}">

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f:{\mathcal {X}}\rightarrow {\mathcal {Y}}^{T}}</annotation>

</semantics>

</math></span><img src="./57ea1926ff77663fd7ec8af90eeedb8009fc67e2.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:11.786ex; height:3.009ex;" alt="{\displaystyle f:{\mathcal {X}}\rightarrow {\mathcal {Y}}^{T}}" loading="lazy"></span> having components <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f_{t}:{\mathcal {X}}\rightarrow {\mathcal {Y}}}">

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f_{t}:{\mathcal {X}}\rightarrow {\mathcal {Y}}}</annotation>

</semantics>

</math></span><img src="./341e98d0ac9360d80811760d1d7862fd19eab37d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:11.051ex; height:2.509ex;" alt="{\displaystyle f_{t}:{\mathcal {X}}\rightarrow {\mathcal {Y}}}" loading="lazy"></span>.

</p><p>The reproducing kernel for the space <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {H}}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {H}}}</annotation>

</semantics>

</math></span><img src="./19ef4c7b923a5125ac91aa491838a95ee15b804f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.964ex; height:2.176ex;" alt="{\displaystyle {\mathcal {H}}}" loading="lazy"></span> of functions <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f:{\mathcal {X}}\rightarrow \mathbb {R} ^{T}}">

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f:{\mathcal {X}}\rightarrow \mathbb {R} ^{T}}</annotation>

</semantics>

</math></span><img src="./090f479500ff878a0955fb00ffe5a1857ff9bb44.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:11.773ex; height:3.009ex;" alt="{\displaystyle f:{\mathcal {X}}\rightarrow \mathbb {R} ^{T}}" loading="lazy"></span> is a symmetric matrix-valued function <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \Gamma :{\mathcal {X}}\times {\mathcal {X}}\rightarrow \mathbb {R} ^{T\times T}}">

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \Gamma :{\mathcal {X}}\times {\mathcal {X}}\rightarrow \mathbb {R} ^{T\times T}}</annotation>

</semantics>

</math></span><img src="./5456b541cd1042d3643d97c931a6ceddb37f417a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:19.098ex; height:2.676ex;" alt="{\displaystyle \Gamma :{\mathcal {X}}\times {\mathcal {X}}\rightarrow \mathbb {R} ^{T\times T}}" loading="lazy"></span> , such that <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \Gamma (\cdot ,x)c\in {\mathcal {H}}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \Gamma (\cdot ,x)c\in {\mathcal {H}}}</annotation>

</semantics>

</math></span><img src="./e35c3372bda118c3a7ba5825c3845d589a2034f4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:12.084ex; height:2.843ex;" alt="{\displaystyle \Gamma (\cdot ,x)c\in {\mathcal {H}}}" loading="lazy"></span> and the following reproducing property holds:

</p>

<msub>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \langle f(x),c\rangle _{\mathbb {R} ^{T}}=\langle f,\Gamma (x,\cdot )c\rangle _{\mathcal {H}}}</annotation>

</semantics>

</math></span></span></td> <td></td> <td class="nowrap"><span id="math_2" class="reference nourlexpansion" style="font-weight:bold;">2</span></td></tr></tbody></table><p> The reproducing kernel gives rise to a representer theorem showing that any solution to equation <b><a href="#math_1">1</a></b> has the form:

</p><table role="presentation" class="numblk" style="margin-left: 1.6em;"><tbody><tr><td class="nowrap"><span class="mwe-math-element mwe-math-element-block"><span class="mwe-math-mathml-display mwe-math-mathml-a11y" style="display: none;"><math display="block" xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f(x)=\sum _{t=1}^{T}\sum _{i=1}^{n_{t}}\Gamma (x,x_{i}^{t})c_{i}^{t}}">

</mrow>

</mrow>

</munderover>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f(x)=\sum _{t=1}^{T}\sum _{i=1}^{n_{t}}\Gamma (x,x_{i}^{t})c_{i}^{t}}</annotation>

</semantics>

</math></span></span></td> <td></td> <td class="nowrap"><span id="math_3" class="reference nourlexpansion" style="font-weight:bold;">3</span></td></tr></tbody></table>

<div class="mw-heading mw-heading4"><h4 id="Separable_kernels">Separable kernels</h4></div>

<p>The form of the kernel <span class="texhtml">Γ</span> induces both the representation of the <a href="Feature_space" class="mw-redirect" title="Feature space">feature space</a> and structures the output across tasks. A natural simplification is to choose a <i>separable kernel,</i> which factors into separate kernels on the input space <span class="mathcal" style="font-family: 'Lucida Calligraphy', 'Monotype Corsiva', 'URW Chancery L', 'Apple Chancery', 'Tex Gyre Chorus', cursive, serif;">X</span> and on the tasks <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \{1,...,T\}}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \{1,...,T\}}</annotation>

</semantics>

</math></span><img src="./3cefbaa24310661c93c2174bc4498c8cd8573c6c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:10.293ex; height:2.843ex;" alt="{\displaystyle \{1,...,T\}}" loading="lazy"></span>. In this case the kernel relating scalar components <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f_{t}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f_{t}}</annotation>

</semantics>

</math></span><img src="./874c306411e808e8191e8aeb95e3440e1c68d6e9.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.965ex; height:2.509ex;" alt="{\displaystyle f_{t}}" loading="lazy"></span> and <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f_{s}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f_{s}}</annotation>

</semantics>

</math></span><img src="./34f933ad7a8dc310b3fa8e9f7b0b2558cba136db.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.143ex; height:2.509ex;" alt="{\displaystyle f_{s}}" loading="lazy"></span> is given by <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\textstyle \gamma ((x_{i},t),(x_{j},s))=k(x_{i},x_{j})k_{T}(s,t)=k(x_{i},x_{j})A_{s,t}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\textstyle \gamma ((x_{i},t),(x_{j},s))=k(x_{i},x_{j})k_{T}(s,t)=k(x_{i},x_{j})A_{s,t}}</annotation>

</semantics>

</math></span><img src="./a5b01e93aa80063bd8e1fcd55282128ad35401dd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:50.306ex; height:3.009ex;" alt="{\textstyle \gamma ((x_{i},t),(x_{j},s))=k(x_{i},x_{j})k_{T}(s,t)=k(x_{i},x_{j})A_{s,t}}" loading="lazy"></span>. For vector valued functions <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f\in {\mathcal {H}}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f\in {\mathcal {H}}}</annotation>

</semantics>

</math></span><img src="./2634be9ba2369b9d92861b6d574a471eae58ff81.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:6.083ex; height:2.509ex;" alt="{\displaystyle f\in {\mathcal {H}}}" loading="lazy"></span> we can write <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \Gamma (x_{i},x_{j})=k(x_{i},x_{j})A}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \Gamma (x_{i},x_{j})=k(x_{i},x_{j})A}</annotation>

</semantics>

</math></span><img src="./a21e25ab8f4dd6b2223a1a67daeb920b5597aee0.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:21.93ex; height:3.009ex;" alt="{\displaystyle \Gamma (x_{i},x_{j})=k(x_{i},x_{j})A}" loading="lazy"></span>, where <span class="texhtml mvar" style="font-style:italic;">k</span> is a scalar reproducing kernel, and <span class="texhtml mvar" style="font-style:italic;">A</span> is a symmetric positive semi-definite <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle T\times T}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle T\times T}</annotation>

</semantics>

</math></span><img src="./75afccf98f544674d475650efd94c84e4de2b2ea.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:6.113ex; height:2.176ex;" alt="{\displaystyle T\times T}" loading="lazy"></span> matrix. Henceforth denote <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle S_{+}^{T}=\{{\text{PSD matrices}}\}\subset \mathbb {R} ^{T\times T}}">

</mrow>

</mrow>

</msubsup>

<mtext>PSD matrices</mtext>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle S_{+}^{T}=\{{\text{PSD matrices}}\}\subset \mathbb {R} ^{T\times T}}</annotation>

</semantics>

</math></span><img src="./2eba83926626afb2dfaa42c9b37b43b0a0fd0b86.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:30.735ex; height:3.343ex;" alt="{\displaystyle S_{+}^{T}=\{{\text{PSD matrices}}\}\subset \mathbb {R} ^{T\times T}}" loading="lazy"></span> .

</p><p>This factorization property, separability, implies the input feature space representation does not vary by task. That is, there is no interaction between the input kernel and the task kernel. The structure on tasks is represented solely by <span class="texhtml mvar" style="font-style:italic;">A</span>. Methods for non-separable kernels <span class="texhtml">Γ</span> is a current field of research.

</p><p>For the separable case, the representation theorem is reduced to <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\textstyle f(x)=\sum _{i=1}^{N}k(x,x_{i})Ac_{i}}">

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\textstyle f(x)=\sum _{i=1}^{N}k(x,x_{i})Ac_{i}}</annotation>

</semantics>

</math></span><img src="./bb74aba5a81201bbaea06df76a0e0ba0e63a3505.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:24.32ex; height:3.509ex;" alt="{\textstyle f(x)=\sum _{i=1}^{N}k(x,x_{i})Ac_{i}}" loading="lazy"></span>. The model output on the training data is then <span class="texhtml mvar" style="font-style:italic;">KCA</span> , where <span class="texhtml mvar" style="font-style:italic;">K</span> is the <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle n\times n}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle n\times n}</annotation>

</semantics>

</math></span><img src="./59d2b4cb72e304526cf5b5887147729ea259da78.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:5.63ex; height:1.676ex;" alt="{\displaystyle n\times n}" loading="lazy"></span> empirical kernel matrix with entries <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\textstyle K_{i,j}=k(x_{i},x_{j})}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\textstyle K_{i,j}=k(x_{i},x_{j})}</annotation>

</semantics>

</math></span><img src="./6a385b212e50aad963a5d8fe763b25b803781821.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:15.429ex; height:3.009ex;" alt="{\textstyle K_{i,j}=k(x_{i},x_{j})}" loading="lazy"></span>, and <span class="texhtml mvar" style="font-style:italic;">C</span> is the <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle n\times T}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle n\times T}</annotation>

</semantics>

</math></span><img src="./b3a5c7e570f3e8ad861252a1147af4e6176d143c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:5.871ex; height:2.176ex;" alt="{\displaystyle n\times T}" loading="lazy"></span> matrix of rows <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle c_{i}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle c_{i}}</annotation>

</semantics>

</math></span><img src="./01acb7953ba52c2aa44264b5d0f8fd223aa178a2.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.807ex; height:2.009ex;" alt="{\displaystyle c_{i}}" loading="lazy"></span>.

</p><p>With the separable kernel, equation <b><a href="#math_1">1</a></b> can be rewritten as

</p>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</munder>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{C\in \mathbb {R} ^{n\times T}}V(Y,KCA)+\lambda tr(KCAC^{\top })}</annotation>

</semantics>

</math></span></span></td> <td></td> <td class="nowrap"><span id="math_P" class="reference nourlexpansion" style="font-weight:bold;">P</span></td></tr></tbody></table>

<p>where <span class="texhtml mvar" style="font-style:italic;">V</span> is a (weighted) average of <span class="mathcal" style="font-family: 'Lucida Calligraphy', 'Monotype Corsiva', 'URW Chancery L', 'Apple Chancery', 'Tex Gyre Chorus', cursive, serif;">L</span> applied entry-wise to <span class="texhtml mvar" style="font-style:italic;">Y</span> and <span class="texhtml mvar" style="font-style:italic;">KCA</span>. (The weight is zero if <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Y_{i}^{t}}">

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Y_{i}^{t}}</annotation>

</semantics>

</math></span><img src="./876625b6c522f6a46e12420755217670b71ac2f6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:2.726ex; height:3.176ex;" alt="{\displaystyle Y_{i}^{t}}" loading="lazy"></span> is a missing observation).

</p><p>Note the second term in <b><a href="#math_P">P</a></b> can be derived as follows:

</p>

<mtr>

<mtd>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

<msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<mtext>(bilinearity)</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</msub>

</mtd>

<mtd>

<mtext>(reproducing property)</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}\|f\|_{\mathcal {H}}^{2}&=\left\langle \sum _{i=1}^{n}k(\cdot ,x_{i})Ac_{i},\sum _{j=1}^{n}k(\cdot ,x_{j})Ac_{j}\right\rangle _{\mathcal {H}}\\&=\sum _{i,j=1}^{n}\langle k(\cdot ,x_{i})Ac_{i},k(\cdot ,x_{j})Ac_{j}\rangle _{\mathcal {H}}&{\text{(bilinearity)}}\\&=\sum _{i,j=1}^{n}\langle k(x_{i},x_{j})Ac_{i},c_{j}\rangle _{\mathbb {R} ^{T}}&{\text{(reproducing property)}}\\&=\sum _{i,j=1}^{n}k(x_{i},x_{j})c_{i}^{\top }Ac_{j}=tr(KCAC^{\top })\end{aligned}}}</annotation>

</semantics>

</math></span><img src="./d042356e34533982ba4a70261644d8d60b7ab09c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -14.228ex; margin-bottom: -0.277ex; width:71.74ex; height:30.176ex;" alt="{\displaystyle {\begin{aligned}\|f\|_{\mathcal {H}}^{2}&=\left\langle \sum _{i=1}^{n}k(\cdot ,x_{i})Ac_{i},\sum _{j=1}^{n}k(\cdot ,x_{j})Ac_{j}\right\rangle _{\mathcal {H}}\\&=\sum _{i,j=1}^{n}\langle k(\cdot ,x_{i})Ac_{i},k(\cdot ,x_{j})Ac_{j}\rangle _{\mathcal {H}}&{\text{(bilinearity)}}\\&=\sum _{i,j=1}^{n}\langle k(x_{i},x_{j})Ac_{i},c_{j}\rangle _{\mathbb {R} ^{T}}&{\text{(reproducing property)}}\\&=\sum _{i,j=1}^{n}k(x_{i},x_{j})c_{i}^{\top }Ac_{j}=tr(KCAC^{\top })\end{aligned}}}" loading="lazy"></span></dd></dl>

<div class="mw-heading mw-heading4"><h4 id="Known_task_structure">Known task structure</h4></div>

<div class="mw-heading mw-heading5"><h5 id="Task_structure_representations">Task structure representations</h5></div>

<p>There are three largely equivalent ways to represent task structure: through a regularizer; through an output metric, and through an output mapping.

</p>

/* start https://en.wikipedia.org/ */

.mw-parser-output .math_theorem{margin:1em 2em;padding:0.5em 1em 0.4em;border:1px solid #aaa;overflow:hidden}@media(max-width:500px){.mw-parser-output .math_theorem{margin:1em 0em;padding:0.5em 0.5em 0.4em}}

/* end https://en.wikipedia.org/ */

</style><div class="math_theorem" style="">

<p><strong class="theorem-name">Regularizer</strong><span class="theoreme-tiret">—</span>With the separable kernel, it can be shown (below) that <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\textstyle ||f||_{\mathcal {H}}^{2}=\sum _{s,t=1}^{T}A_{t,s}^{\dagger }\langle f_{s},f_{t}\rangle _{{\mathcal {H}}_{k}}}">

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\textstyle ||f||_{\mathcal {H}}^{2}=\sum _{s,t=1}^{T}A_{t,s}^{\dagger }\langle f_{s},f_{t}\rangle _{{\mathcal {H}}_{k}}}</annotation>

</semantics>

</math></span><img src="./52dd02e5cf9b9474c2d301a0ebc452ac087d230f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.171ex; width:28.81ex; height:3.676ex;" alt="{\textstyle ||f||_{\mathcal {H}}^{2}=\sum _{s,t=1}^{T}A_{t,s}^{\dagger }\langle f_{s},f_{t}\rangle _{{\mathcal {H}}_{k}}}" loading="lazy"></span>, where <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle A_{t,s}^{\dagger }}">

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle A_{t,s}^{\dagger }}</annotation>

</semantics>

</math></span><img src="./122b6d8393c9440e345161fe1332ba7594684380.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.171ex; width:3.798ex; height:3.676ex;" alt="{\displaystyle A_{t,s}^{\dagger }}" loading="lazy"></span> is the <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle t,s}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle t,s}</annotation>

</semantics>

</math></span><img src="./38a01ccb6fb590c7debcbe9de8e199c1dcc26f78.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.964ex; height:2.343ex;" alt="{\displaystyle t,s}" loading="lazy"></span> element of the pseudoinverse of <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle A}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle A}</annotation>

</semantics>

</math></span><img src="./7daff47fa58cdfd29dc333def748ff5fa4c923e3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.743ex; height:2.176ex;" alt="{\displaystyle A}" loading="lazy"></span>, and <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {H}}_{k}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {H}}_{k}}</annotation>

</semantics>

</math></span><img src="./f6681a90bfa2d32c42916295433d343105977884.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.052ex; height:2.509ex;" alt="{\displaystyle {\mathcal {H}}_{k}}" loading="lazy"></span> is the RKHS based on the scalar kernel <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle k}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle k}</annotation>

</semantics>

</math></span><img src="./c3c9a2c7b599b37105512c5d570edc034056dd40.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.211ex; height:2.176ex;" alt="{\displaystyle k}" loading="lazy"></span>, and <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\textstyle f_{t}(x)=\sum _{i=1}^{n}k(x,x_{i})A_{t}^{\top }c_{i}}">

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\textstyle f_{t}(x)=\sum _{i=1}^{n}k(x,x_{i})A_{t}^{\top }c_{i}}</annotation>

</semantics>

</math></span><img src="./3b6034509ef70fbe32dc99d1ee580411a39e326e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:26.518ex; height:3.176ex;" alt="{\textstyle f_{t}(x)=\sum _{i=1}^{n}k(x,x_{i})A_{t}^{\top }c_{i}}" loading="lazy"></span>. This formulation shows that <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle A_{t,s}^{\dagger }}">

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle A_{t,s}^{\dagger }}</annotation>

</semantics>

</math></span><img src="./122b6d8393c9440e345161fe1332ba7594684380.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.171ex; width:3.798ex; height:3.676ex;" alt="{\displaystyle A_{t,s}^{\dagger }}" loading="lazy"></span> controls the weight of the penalty associated with <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\textstyle \langle f_{s},f_{t}\rangle _{{\mathcal {H}}_{k}}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\textstyle \langle f_{s},f_{t}\rangle _{{\mathcal {H}}_{k}}}</annotation>

</semantics>

</math></span><img src="./4ba98d0c99cd6c97d645d9da1ee045454a202cfd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:9.431ex; height:3.009ex;" alt="{\textstyle \langle f_{s},f_{t}\rangle _{{\mathcal {H}}_{k}}}" loading="lazy"></span>. (Note that <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\textstyle \langle f_{s},f_{t}\rangle _{{\mathcal {H}}_{k}}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\textstyle \langle f_{s},f_{t}\rangle _{{\mathcal {H}}_{k}}}</annotation>

</semantics>

</math></span><img src="./4ba98d0c99cd6c97d645d9da1ee045454a202cfd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:9.431ex; height:3.009ex;" alt="{\textstyle \langle f_{s},f_{t}\rangle _{{\mathcal {H}}_{k}}}" loading="lazy"></span> arises from <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\textstyle ||f_{t}||_{{\mathcal {H}}_{k}}=\langle f_{t},f_{t}\rangle _{{\mathcal {H}}_{k}}}">

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\textstyle ||f_{t}||_{{\mathcal {H}}_{k}}=\langle f_{t},f_{t}\rangle _{{\mathcal {H}}_{k}}}</annotation>

</semantics>

</math></span><img src="./e363a0b9cd79815643771c6dd61836025aec4e1d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.338ex; width:19.385ex; height:3.343ex;" alt="{\textstyle ||f_{t}||_{{\mathcal {H}}_{k}}=\langle f_{t},f_{t}\rangle _{{\mathcal {H}}_{k}}}" loading="lazy"></span>.)

</p>

/* start https://en.wikipedia.org/ */

.mw-parser-output .math_proof{border:thin solid #aaa;margin:1em 2em;padding:0.5em 1em 0.4em}@media(max-width:500px){.mw-parser-output .math_proof{margin:1em 0;padding:0.5em 0.5em 0.4em}}

/* end https://en.wikipedia.org/ */

</style><div class="math_proof" style=""><strong>Proof</strong>

<mtr>

<mtd>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

<msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msubsup>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msubsup>

</mrow>

</mrow>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msubsup>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msub>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msub>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}\|f\|_{\mathcal {H}}^{2}&=\left\langle \sum _{i=1}^{n}\gamma ((x_{i},t_{i}),\cdot )c_{i}^{t_{i}},\sum _{j=1}^{n}\gamma ((x_{j},t_{j}),\cdot )c_{j}^{t_{j}}\right\rangle _{\mathcal {H}}\\&=\sum _{i,j=1}^{n}c_{i}^{t_{i}}c_{j}^{t_{j}}\gamma ((x_{i},t_{i}),(x_{j},t_{j}))\\&=\sum _{i,j=1}^{n}\sum _{s,t=1}^{T}c_{i}^{t}c_{j}^{s}k(x_{i},x_{j})A_{s,t}\\&=\sum _{i,j=1}^{n}k(x_{i},x_{j})\langle c_{i},Ac_{j}\rangle _{\mathbb {R} ^{T}}\\&=\sum _{i,j=1}^{n}k(x_{i},x_{j})\langle c_{i},AA^{\dagger }Ac_{j}\rangle _{\mathbb {R} ^{T}}\\&=\sum _{i,j=1}^{n}k(x_{i},x_{j})\langle Ac_{i},A^{\dagger }Ac_{j}\rangle _{\mathbb {R} ^{T}}\\&=\sum _{i,j=1}^{n}\sum _{s,t=1}^{T}(Ac_{i})^{t}(Ac_{j})^{s}k(x_{i},x_{j})A_{s,t}^{\dagger }\\&=\sum _{s,t=1}^{T}A_{s,t}^{\dagger }\langle \sum _{i=1}^{n}k(x_{i},\cdot )(Ac_{i})^{t},\sum _{j=1}^{n}k(x_{j},\cdot )(Ac_{j})^{s}\rangle _{{\mathcal {H}}_{k}}\\&=\sum _{s,t=1}^{T}A_{s,t}^{\dagger }\langle f_{t},f_{s}\rangle _{{\mathcal {H}}_{k}}\end{aligned}}}</annotation>

</semantics>

</math></span><img src="./8342ed67bdc3162e517540acb663ec581ed2140e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -33.565ex; margin-bottom: -0.272ex; width:56.418ex; height:68.843ex;" alt="{\displaystyle {\begin{aligned}\|f\|_{\mathcal {H}}^{2}&=\left\langle \sum _{i=1}^{n}\gamma ((x_{i},t_{i}),\cdot )c_{i}^{t_{i}},\sum _{j=1}^{n}\gamma ((x_{j},t_{j}),\cdot )c_{j}^{t_{j}}\right\rangle _{\mathcal {H}}\\&=\sum _{i,j=1}^{n}c_{i}^{t_{i}}c_{j}^{t_{j}}\gamma ((x_{i},t_{i}),(x_{j},t_{j}))\\&=\sum _{i,j=1}^{n}\sum _{s,t=1}^{T}c_{i}^{t}c_{j}^{s}k(x_{i},x_{j})A_{s,t}\\&=\sum _{i,j=1}^{n}k(x_{i},x_{j})\langle c_{i},Ac_{j}\rangle _{\mathbb {R} ^{T}}\\&=\sum _{i,j=1}^{n}k(x_{i},x_{j})\langle c_{i},AA^{\dagger }Ac_{j}\rangle _{\mathbb {R} ^{T}}\\&=\sum _{i,j=1}^{n}k(x_{i},x_{j})\langle Ac_{i},A^{\dagger }Ac_{j}\rangle _{\mathbb {R} ^{T}}\\&=\sum _{i,j=1}^{n}\sum _{s,t=1}^{T}(Ac_{i})^{t}(Ac_{j})^{s}k(x_{i},x_{j})A_{s,t}^{\dagger }\\&=\sum _{s,t=1}^{T}A_{s,t}^{\dagger }\langle \sum _{i=1}^{n}k(x_{i},\cdot )(Ac_{i})^{t},\sum _{j=1}^{n}k(x_{j},\cdot )(Ac_{j})^{s}\rangle _{{\mathcal {H}}_{k}}\\&=\sum _{s,t=1}^{T}A_{s,t}^{\dagger }\langle f_{t},f_{s}\rangle _{{\mathcal {H}}_{k}}\end{aligned}}}" loading="lazy"></span>

</p>

</div>

<p><strong class="theorem-name">Output metric</strong><span class="theoreme-tiret">—</span>an alternative output metric on <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {Y}}^{T}}">

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {Y}}^{T}}</annotation>

</semantics>

</math></span><img src="./5d5f8f6c006526d1df80c43682e3c3323d364d31.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.081ex; height:3.009ex;" alt="{\displaystyle {\mathcal {Y}}^{T}}" loading="lazy"></span> can be induced by the inner product <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \langle y_{1},y_{2}\rangle _{\Theta }=\langle y_{1},\Theta y_{2}\rangle _{\mathbb {R} ^{T}}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \langle y_{1},y_{2}\rangle _{\Theta }=\langle y_{1},\Theta y_{2}\rangle _{\mathbb {R} ^{T}}}</annotation>

</semantics>

</math></span><img src="./fb29a4b44e313052809c2e18bb91b34f1b320718.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:23.4ex; height:3.009ex;" alt="{\displaystyle \langle y_{1},y_{2}\rangle _{\Theta }=\langle y_{1},\Theta y_{2}\rangle _{\mathbb {R} ^{T}}}" loading="lazy"></span>. With the squared loss there is an equivalence between the separable kernels <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle k(\cdot ,\cdot )I_{T}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle k(\cdot ,\cdot )I_{T}}</annotation>

</semantics>

</math></span><img src="./98e284378cc756262fc06d0f084abe1d13123e9b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:7.761ex; height:2.843ex;" alt="{\displaystyle k(\cdot ,\cdot )I_{T}}" loading="lazy"></span> under the alternative metric, and <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle k(\cdot ,\cdot )\Theta }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle k(\cdot ,\cdot )\Theta }</annotation>

</semantics>

</math></span><img src="./33b9e10fe85f0eb652324f6010287ad8896e9fb4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:7.156ex; height:2.843ex;" alt="{\displaystyle k(\cdot ,\cdot )\Theta }" loading="lazy"></span>, under the canonical metric.

</p>

</div>

<p><strong class="theorem-name">Output mapping</strong><span class="theoreme-tiret">—</span>Outputs can be mapped as <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L:{\mathcal {Y}}^{T}\rightarrow {\mathcal {\tilde {Y}}}}">

<msup>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L:{\mathcal {Y}}^{T}\rightarrow {\mathcal {\tilde {Y}}}}</annotation>

</semantics>

</math></span><img src="./fe8f1d2edc0f308070de3ba4286458c522049ab6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:11.889ex; height:3.009ex;" alt="{\displaystyle L:{\mathcal {Y}}^{T}\rightarrow {\mathcal {\tilde {Y}}}}" loading="lazy"></span> to a higher dimensional space to encode complex structures such as trees, graphs and strings. For linear maps <span class="texhtml mvar" style="font-style:italic;">L</span>, with appropriate choice of separable kernel, it can be shown that <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle A=L^{\top }L}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle A=L^{\top }L}</annotation>

</semantics>

</math></span><img src="./673d8ba1fe8e82e1f58843f3b584a4ccda4aa6d3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:9.518ex; height:2.676ex;" alt="{\displaystyle A=L^{\top }L}" loading="lazy"></span>.

</p>

</div>

<div class="mw-heading mw-heading5"><h5 id="Task_structure_examples">Task structure examples</h5></div>

<p>Via the regularizer formulation, one can represent a variety of task structures easily.

</p>

<ul><li>Letting <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\textstyle A^{\dagger }=\gamma I_{T}+(\gamma -\lambda ){\frac {1}{T}}\mathbf {1} \mathbf {1} ^{\top }}">

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\textstyle A^{\dagger }=\gamma I_{T}+(\gamma -\lambda ){\frac {1}{T}}\mathbf {1} \mathbf {1} ^{\top }}</annotation>

</semantics>

</math></span><img src="./35db9e0c138a087f6f78dc396338d58ca1086da1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.171ex; width:25.763ex; height:3.509ex;" alt="{\textstyle A^{\dagger }=\gamma I_{T}+(\gamma -\lambda ){\frac {1}{T}}\mathbf {1} \mathbf {1} ^{\top }}" loading="lazy"></span> (where <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle I_{T}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle I_{T}}</annotation>

</semantics>

</math></span><img src="./0555645749867b84164d65d1ac50ee8087220601.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.412ex; height:2.509ex;" alt="{\displaystyle I_{T}}" loading="lazy"></span> is the <i>T</i>x<i>T</i> identity matrix, and <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\textstyle \mathbf {1} \mathbf {1} ^{\top }}">

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\textstyle \mathbf {1} \mathbf {1} ^{\top }}</annotation>

</semantics>

</math></span><img src="./6a9c020d3998f30ea88ae32c79fc11ec1a0080f3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:4.184ex; height:2.509ex;" alt="{\textstyle \mathbf {1} \mathbf {1} ^{\top }}" loading="lazy"></span> is the <i>T</i>x<i>T</i> matrix of ones) is equivalent to letting <span class="texhtml">Γ</span> control the variance <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\textstyle \sum _{t}||f_{t}-{\bar {f}}||_{{\mathcal {H}}_{k}}}">

</mrow>

</munder>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mover>

</mover>

</mrow>

</mrow>

<msub>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\textstyle \sum _{t}||f_{t}-{\bar {f}}||_{{\mathcal {H}}_{k}}}</annotation>

</semantics>

</math></span><img src="./91b35ffa7c2ace580ff1659a3fc8be10e072b5fa.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.338ex; width:15.239ex; height:3.676ex;" alt="{\textstyle \sum _{t}||f_{t}-{\bar {f}}||_{{\mathcal {H}}_{k}}}" loading="lazy"></span> of tasks from their mean <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\textstyle {\frac {1}{T}}\sum _{t}f_{t}}">

<mfrac>

</mfrac>

</mrow>

</mrow>

</munder>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\textstyle {\frac {1}{T}}\sum _{t}f_{t}}</annotation>

</semantics>

</math></span><img src="./391d8fd965a372633e167010c0468875356c3547.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.171ex; width:8.012ex; height:3.509ex;" alt="{\textstyle {\frac {1}{T}}\sum _{t}f_{t}}" loading="lazy"></span>. For example, blood levels of some biomarker may be taken on <span class="texhtml mvar" style="font-style:italic;">T</span> patients at <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle n_{t}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle n_{t}}</annotation>

</semantics>

</math></span><img src="./27f8546859bf39892f3c245d8622e3e9cee286b9.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.221ex; height:2.009ex;" alt="{\displaystyle n_{t}}" loading="lazy"></span> time points during the course of a day and interest may lie in regularizing the variance of the predictions across patients.</li>

<li>Letting <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle A^{\dagger }=\alpha I_{T}+(\alpha -\lambda )M}">

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle A^{\dagger }=\alpha I_{T}+(\alpha -\lambda )M}</annotation>

</semantics>

</math></span><img src="./763a44246ff2b52ad0e151d6a5bcb3172c622766.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:22.479ex; height:3.176ex;" alt="{\displaystyle A^{\dagger }=\alpha I_{T}+(\alpha -\lambda )M}" loading="lazy"></span> , where <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle M_{t,s}={\frac {1}{|G_{r}|}}\mathbb {I} (t,s\in G_{r})}">

<msub>

</mrow>

</msub>

<mfrac>

<mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle M_{t,s}={\frac {1}{|G_{r}|}}\mathbb {I} (t,s\in G_{r})}</annotation>

</semantics>

</math></span><img src="./b83f854bde1aff94b7585007227ffbccc03978ca.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.671ex; width:23.656ex; height:6.009ex;" alt="{\displaystyle M_{t,s}={\frac {1}{|G_{r}|}}\mathbb {I} (t,s\in G_{r})}" loading="lazy"></span> is equivalent to letting <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \alpha }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \alpha }</annotation>

</semantics>

</math></span><img src="./b79333175c8b3f0840bfb4ec41b8072c83ea88d3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.488ex; height:1.676ex;" alt="{\displaystyle \alpha }" loading="lazy"></span> control the variance measured with respect to a group mean: <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \sum _{r}\sum _{t\in G_{r}}||f_{t}-{\frac {1}{|G_{r}|}}\sum _{s\in G_{r})}f_{s}||}">

</mrow>

</munder>

<msub>

</mrow>

</msub>

</mrow>

</munder>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mfrac>

<mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</munder>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \sum _{r}\sum _{t\in G_{r}}||f_{t}-{\frac {1}{|G_{r}|}}\sum _{s\in G_{r})}f_{s}||}</annotation>

</semantics>

</math></span><img src="./27231b77c66808950385c875c481d468a24d1aa6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.505ex; width:27.682ex; height:6.843ex;" alt="{\displaystyle \sum _{r}\sum _{t\in G_{r}}||f_{t}-{\frac {1}{|G_{r}|}}\sum _{s\in G_{r})}f_{s}||}" loading="lazy"></span>. (Here <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle |G_{r}|}">

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle |G_{r}|}</annotation>

</semantics>

</math></span><img src="./d98f5c934d3e59f932b6df3e20db8c3d3e3279d9.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:4.094ex; height:2.843ex;" alt="{\displaystyle |G_{r}|}" loading="lazy"></span> the cardinality of group r, and <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbb {I} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbb {I} }</annotation>

</semantics>

</math></span><img src="./8205f06e0d279689ed04a1ac04a3d9c249c637df.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:0.905ex; height:2.176ex;" alt="{\displaystyle \mathbb {I} }" loading="lazy"></span> is the indicator function). For example, people in different political parties (groups) might be regularized together with respect to predicting the favorability rating of a politician. Note that this penalty reduces to the first when all tasks are in the same group.</li>

<li>Letting <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle A^{\dagger }=\delta I_{T}+(\delta -\lambda )L}">

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle A^{\dagger }=\delta I_{T}+(\delta -\lambda )L}</annotation>

</semantics>

</math></span><img src="./2a1286bcd5cc33d4f071b9358217a44fb592ca25.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:20.742ex; height:3.176ex;" alt="{\displaystyle A^{\dagger }=\delta I_{T}+(\delta -\lambda )L}" loading="lazy"></span>, where <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L=D-M}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L=D-M}</annotation>

</semantics>

</math></span><img src="./cc65c4871c9cd644b779078cc49c8b24958a83d4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.505ex; width:11.888ex; height:2.343ex;" alt="{\displaystyle L=D-M}" loading="lazy"></span> is the <a href="Laplacian_matrix" title="Laplacian matrix">Laplacian</a> for the graph with <a href="Adjacency_matrix" title="Adjacency matrix">adjacency matrix</a> <i>M</i> giving pairwise similarities of tasks. This is equivalent to giving a larger penalty to the distance separating tasks <i>t</i> and <i>s</i> when they are more similar (according to the weight <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle M_{t,s}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle M_{t,s}}</annotation>

</semantics>

</math></span><img src="./230599ef12e815283b111a5878a9edbcdc5608a7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:4.308ex; height:2.843ex;" alt="{\displaystyle M_{t,s}}" loading="lazy"></span>,) i.e. <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \delta }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \delta }</annotation>

</semantics>

</math></span><img src="./c5321cfa797202b3e1f8620663ff43c4660ea03a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.049ex; height:2.343ex;" alt="{\displaystyle \delta }" loading="lazy"></span> regularizes <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \sum _{t,s}||f_{t}-f_{s}||_{{\mathcal {H}}_{k}}^{2}M_{t,s}}">

</mrow>

</munder>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \sum _{t,s}||f_{t}-f_{s}||_{{\mathcal {H}}_{k}}^{2}M_{t,s}}</annotation>

</semantics>

</math></span><img src="./c6599c8dfbd65ecd505e6c0544aec84e14da0512.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.338ex; width:20.066ex; height:5.843ex;" alt="{\displaystyle \sum _{t,s}||f_{t}-f_{s}||_{{\mathcal {H}}_{k}}^{2}M_{t,s}}" loading="lazy"></span>.</li>

<li>All of the above choices of A also induce the additional regularization term <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\textstyle \lambda \sum _{t}||f||_{{\mathcal {H}}_{k}}^{2}}">

</mrow>

</munder>

</mrow>

</mrow>

</mrow>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\textstyle \lambda \sum _{t}||f||_{{\mathcal {H}}_{k}}^{2}}</annotation>

</semantics>

</math></span><img src="./4a553d498b16a676a3da8a1381dbce5a9bd2a888.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.338ex; width:11.756ex; height:3.843ex;" alt="{\textstyle \lambda \sum _{t}||f||_{{\mathcal {H}}_{k}}^{2}}" loading="lazy"></span> which penalizes complexity in f more broadly.</li></ul>

<div class="mw-heading mw-heading4"><h4 id="Learning_tasks_together_with_their_structure">Learning tasks together with their structure</h4></div>

<p>Learning problem <b><a href="#math_P">P</a></b> can be generalized to admit learning task matrix A as follows:

</p>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

</msubsup>

</mrow>

</munder>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{C\in \mathbb {R} ^{n\times T},A\in S_{+}^{T}}V(Y,KCA)+\lambda tr(KCAC^{\top })+F(A)}</annotation>

</semantics>

</math></span></span></td> <td></td> <td class="nowrap"><span id="math_Q" class="reference nourlexpansion" style="font-weight:bold;">Q</span></td></tr></tbody></table>

<p>Choice of <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle F:S_{+}^{T}\rightarrow \mathbb {R} _{+}}">

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle F:S_{+}^{T}\rightarrow \mathbb {R} _{+}}</annotation>

</semantics>

</math></span><img src="./0e98987663a0216d4f9a6b085520bd19ea504755.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:13.417ex; height:3.176ex;" alt="{\displaystyle F:S_{+}^{T}\rightarrow \mathbb {R} _{+}}" loading="lazy"></span> must be designed to learn matrices <i>A</i> of a given type. See "Special cases" below.

</p>

<div class="mw-heading mw-heading5"><h5 id="Optimization_of_Q">Optimization of <b><a href="#math_Q">Q</a></b></h5></div>

<p>Restricting to the case of <a href="Convex_optimization" title="Convex optimization">convex</a> losses and <a href="Coercive_function" title="Coercive function">coercive</a> penalties Ciliberto <i>et al.</i> have shown that although <b><a href="#math_Q">Q</a></b> is not convex jointly in <i>C</i> and <i>A,</i> a related problem is jointly convex.

</p><p>Specifically on the convex set <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {C}}=\{(C,A)\in \mathbb {R} ^{n\times T}\times S_{+}^{T}|Range(C^{\top }KC)\subseteq Range(A)\}}">

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

</msubsup>

</mrow>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {C}}=\{(C,A)\in \mathbb {R} ^{n\times T}\times S_{+}^{T}|Range(C^{\top }KC)\subseteq Range(A)\}}</annotation>

</semantics>

</math></span><img src="./a921fd71375fcc43e46260ad991ce2f53a5b423c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:56.388ex; height:3.343ex;" alt="{\displaystyle {\mathcal {C}}=\{(C,A)\in \mathbb {R} ^{n\times T}\times S_{+}^{T}|Range(C^{\top }KC)\subseteq Range(A)\}}" loading="lazy"></span>, the equivalent problem

</p>

</mrow>

</munder>

<msup>

</mrow>

</msup>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{C,A\in {\mathcal {C}}}V(Y,KC)+\lambda tr(A^{\dagger }C^{\top }KC)+F(A)}</annotation>

</semantics>

</math></span></span></td> <td></td> <td class="nowrap"><span id="math_R" class="reference nourlexpansion" style="font-weight:bold;">R</span></td></tr></tbody></table>

<p>is convex with the same minimum value. And if <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle (C_{R},A_{R})}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle (C_{R},A_{R})}</annotation>

</semantics>

</math></span><img src="./148ae4503b82d5600af7d72bdc6274f5ff914a67.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:9.207ex; height:2.843ex;" alt="{\displaystyle (C_{R},A_{R})}" loading="lazy"></span> is a minimizer for <b><a href="#math_R">R</a></b> then <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle (C_{R}A_{R}^{\dagger },A_{R})}">

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle (C_{R}A_{R}^{\dagger },A_{R})}</annotation>

</semantics>

</math></span><img src="./26f2c99b3b9d94138391d870fb78378f1658e868.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:12.43ex; height:3.509ex;" alt="{\displaystyle (C_{R}A_{R}^{\dagger },A_{R})}" loading="lazy"></span> is a minimizer for <b><a href="#math_Q">Q</a></b>.

</p><p><b><a href="#math_R">R</a></b> may be solved by a barrier method on a closed set by introducing the following perturbation:

</p>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

</msubsup>

</mrow>

</munder>

<msup>

</mrow>

</msup>

<msup>

</mrow>

</msup>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{C\in \mathbb {R} ^{n\times T},A\in S_{+}^{T}}V(Y,KC)+\lambda tr(A^{\dagger }(C^{\top }KC+\delta ^{2}I_{T}))+F(A)}</annotation>

</semantics>

</math></span></span></td> <td></td> <td class="nowrap"><span id="math_S" class="reference nourlexpansion" style="font-weight:bold;">S</span></td></tr></tbody></table>

<p>The perturbation via the barrier <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \delta ^{2}tr(A^{\dagger })}">

<msup>

</mrow>

</msup>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \delta ^{2}tr(A^{\dagger })}</annotation>

</semantics>

</math></span><img src="./eaefbf160efb72a42791114c4e2d683b9e1f8d25.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:8.511ex; height:3.176ex;" alt="{\displaystyle \delta ^{2}tr(A^{\dagger })}" loading="lazy"></span> forces the objective functions to be equal to <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle +\infty }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle +\infty }</annotation>

</semantics>

</math></span><img src="./bddbb0e4420a7e744cf71bd71216e11b0bf88831.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.505ex; width:4.132ex; height:2.176ex;" alt="{\displaystyle +\infty }" loading="lazy"></span> on the boundary of <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle R^{n\times T}\times S_{+}^{T}}">

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R^{n\times T}\times S_{+}^{T}}</annotation>

</semantics>

</math></span><img src="./76eaddb313d405d693815c796e9865280c4939b2.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:11.194ex; height:3.343ex;" alt="{\displaystyle R^{n\times T}\times S_{+}^{T}}" loading="lazy"></span> .

</p><p><b><a href="#math_S">S</a></b> can be solved with a block coordinate descent method, alternating in <i>C</i> and <i>A.</i> This results in a sequence of minimizers <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle (C_{m},A_{m})}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle (C_{m},A_{m})}</annotation>

</semantics>

</math></span><img src="./a29130af0c1a741b4fff366a281ffe36dff4e3f1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:9.598ex; height:2.843ex;" alt="{\displaystyle (C_{m},A_{m})}" loading="lazy"></span> in <b><a href="#math_S">S</a></b> that converges to the solution in <b><a href="#math_R">R</a></b> as <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \delta _{m}\rightarrow 0}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \delta _{m}\rightarrow 0}</annotation>

</semantics>

</math></span><img src="./6b44292b6fef7d7eaff9f197b7e45980a8f772ca.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:7.484ex; height:2.676ex;" alt="{\displaystyle \delta _{m}\rightarrow 0}" loading="lazy"></span>, and hence gives the solution to <b><a href="#math_Q">Q</a></b>.

</p>

<div class="mw-heading mw-heading5"><h5 id="Special_cases">Special cases</h5></div>

<p><b><a href="Regularization_by_spectral_filtering" title="Regularization by spectral filtering">Spectral penalties</a></b> - Dinnuzo <i>et al</i><sup id="cite_ref-43" class="reference"><a href="#cite_note-43"><span class="cite-bracket">[</span>43<span class="cite-bracket">]</span></a></sup> suggested setting <i>F</i> as the Frobenius norm <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\sqrt {tr(A^{\top }A)}}}">

<msqrt>

<msup>

</mrow>

</msup>

</msqrt>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\sqrt {tr(A^{\top }A)}}}</annotation>

</semantics>

</math></span><img src="./be0d1b0abf4e9d14ee3b4ac43856dc50a321887b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.671ex; width:11.018ex; height:4.843ex;" alt="{\displaystyle {\sqrt {tr(A^{\top }A)}}}" loading="lazy"></span>. They optimized <b><a href="#math_Q">Q</a></b> directly using block coordinate descent, not accounting for difficulties at the boundary of <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbb {R} ^{n\times T}\times S_{+}^{T}}">

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbb {R} ^{n\times T}\times S_{+}^{T}}</annotation>

</semantics>

</math></span><img src="./96ae0a8940e9ed5192aad505415781b19f24611d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:11.108ex; height:3.343ex;" alt="{\displaystyle \mathbb {R} ^{n\times T}\times S_{+}^{T}}" loading="lazy"></span>.

</p><p><b>Clustered tasks learning</b> - Jacob <i>et al</i><sup id="cite_ref-44" class="reference"><a href="#cite_note-44"><span class="cite-bracket">[</span>44<span class="cite-bracket">]</span></a></sup> suggested to learn <i>A</i> in the setting where <i>T</i> tasks are organized in <i>R</i> disjoint clusters. In this case let <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle E\in \{0,1\}^{T\times R}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle E\in \{0,1\}^{T\times R}}</annotation>

</semantics>

</math></span><img src="./b6435713ef2b08f94163c37a790d3aea7bf935a7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:14.215ex; height:3.176ex;" alt="{\displaystyle E\in \{0,1\}^{T\times R}}" loading="lazy"></span> be the matrix with <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle E_{t,r}=\mathbb {I} ({\text{task }}t\in {\text{group }}r)}">

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<mtext>group </mtext>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle E_{t,r}=\mathbb {I} ({\text{task }}t\in {\text{group }}r)}</annotation>

</semantics>

</math></span><img src="./09fe80b6a92e475f17fab810567c406f7b2954b2.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:25.475ex; height:3.009ex;" alt="{\displaystyle E_{t,r}=\mathbb {I} ({\text{task }}t\in {\text{group }}r)}" loading="lazy"></span>. Setting <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle M=I-E^{\dagger }E^{T}}">

<msup>

</mrow>

</msup>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle M=I-E^{\dagger }E^{T}}</annotation>

</semantics>

</math></span><img src="./bc0e47603af449c48ddfa91b03c554002d85d291.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.505ex; width:15.492ex; height:2.843ex;" alt="{\displaystyle M=I-E^{\dagger }E^{T}}" loading="lazy"></span>, and <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle U={\frac {1}{T}}\mathbf {11} ^{\top }}">

<mfrac>

</mfrac>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle U={\frac {1}{T}}\mathbf {11} ^{\top }}</annotation>

</semantics>

</math></span><img src="./d966e83fd5e3a450b151da2ba7a47cde0fd5dd1d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.838ex; width:11.538ex; height:5.176ex;" alt="{\displaystyle U={\frac {1}{T}}\mathbf {11} ^{\top }}" loading="lazy"></span>, the task matrix <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle A^{\dagger }}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle A^{\dagger }}</annotation>

</semantics>

</math></span><img src="./253906923e0ca67ef31e4e2d99b9783ba62bebc5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.705ex; height:2.676ex;" alt="{\displaystyle A^{\dagger }}" loading="lazy"></span> can be parameterized as a function of <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle M}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle M}</annotation>

</semantics>

</math></span><img src="./f82cade9898ced02fdd08712e5f0c0151758a0dd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.442ex; height:2.176ex;" alt="{\displaystyle M}" loading="lazy"></span>: <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle A^{\dagger }(M)=\epsilon _{M}U+\epsilon _{B}(M-U)+\epsilon (I-M)}">

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle A^{\dagger }(M)=\epsilon _{M}U+\epsilon _{B}(M-U)+\epsilon (I-M)}</annotation>

</semantics>

</math></span><img src="./f60f839560889015633c97f817560c056283da10.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:40.928ex; height:3.176ex;" alt="{\displaystyle A^{\dagger }(M)=\epsilon _{M}U+\epsilon _{B}(M-U)+\epsilon (I-M)}" loading="lazy"></span> , with terms that penalize the average, between clusters variance and within clusters variance respectively of the task predictions. M is not convex, but there is a convex relaxation <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {S}}_{c}=\{M\in S_{+}^{T}:I-M\in S_{+}^{T}\land tr(M)=r\}}">

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {S}}_{c}=\{M\in S_{+}^{T}:I-M\in S_{+}^{T}\land tr(M)=r\}}</annotation>

</semantics>

</math></span><img src="./868fd14e1119b7178aa20fdfb940faeee9e7bd06.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:43.032ex; height:3.176ex;" alt="{\displaystyle {\mathcal {S}}_{c}=\{M\in S_{+}^{T}:I-M\in S_{+}^{T}\land tr(M)=r\}}" loading="lazy"></span>. In this formulation, <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle F(A)=\mathbb {I} (A(M)\in \{A:M\in {\mathcal {S}}_{C}\})}">

</mrow>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle F(A)=\mathbb {I} (A(M)\in \{A:M\in {\mathcal {S}}_{C}\})}</annotation>

</semantics>

</math></span><img src="./d0a3eb3359028106fcc453c2dc54a6410c9b2239.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:34.119ex; height:2.843ex;" alt="{\displaystyle F(A)=\mathbb {I} (A(M)\in \{A:M\in {\mathcal {S}}_{C}\})}" loading="lazy"></span>.

</p>

<div class="mw-heading mw-heading5"><h5 id="Generalizations">Generalizations</h5></div>

<p><b>Non-convex penalties</b> - Penalties can be constructed such that A is constrained to be a graph Laplacian, or that A has low rank factorization. However these penalties are not convex, and the analysis of the barrier method proposed by Ciliberto et al. does not go through in these cases.

</p><p><b>Non-separable kernels</b> - Separable kernels are limited, in particular they do not account for structures in the interaction space between the input and output domains jointly. Future work is needed to develop models for these kernels.

</p>

<div class="mw-heading mw-heading2"><h2 id="Software_package">Software package</h2></div>

<p>A Matlab package called Multi-Task Learning via StructurAl Regularization (MALSAR) <sup id="cite_ref-45" class="reference"><a href="#cite_note-45"><span class="cite-bracket">[</span>45<span class="cite-bracket">]</span></a></sup> implements the following multi-task learning algorithms: Mean-Regularized Multi-Task Learning,<sup id="cite_ref-46" class="reference"><a href="#cite_note-46"><span class="cite-bracket">[</span>46<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-47" class="reference"><a href="#cite_note-47"><span class="cite-bracket">[</span>47<span class="cite-bracket">]</span></a></sup> Multi-Task Learning with Joint Feature Selection,<sup id="cite_ref-48" class="reference"><a href="#cite_note-48"><span class="cite-bracket">[</span>48<span class="cite-bracket">]</span></a></sup> Robust Multi-Task Feature Learning,<sup id="cite_ref-49" class="reference"><a href="#cite_note-49"><span class="cite-bracket">[</span>49<span class="cite-bracket">]</span></a></sup> Trace-Norm Regularized Multi-Task Learning,<sup id="cite_ref-50" class="reference"><a href="#cite_note-50"><span class="cite-bracket">[</span>50<span class="cite-bracket">]</span></a></sup> Alternating Structural Optimization,<sup id="cite_ref-51" class="reference"><a href="#cite_note-51"><span class="cite-bracket">[</span>51<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-52" class="reference"><a href="#cite_note-52"><span class="cite-bracket">[</span>52<span class="cite-bracket">]</span></a></sup> Incoherent Low-Rank and Sparse Learning,<sup id="cite_ref-53" class="reference"><a href="#cite_note-53"><span class="cite-bracket">[</span>53<span class="cite-bracket">]</span></a></sup> Robust Low-Rank Multi-Task Learning, Clustered Multi-Task Learning,<sup id="cite_ref-54" class="reference"><a href="#cite_note-54"><span class="cite-bracket">[</span>54<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-55" class="reference"><a href="#cite_note-55"><span class="cite-bracket">[</span>55<span class="cite-bracket">]</span></a></sup> Multi-Task Learning with Graph Structures.

</p>

<div class="mw-heading mw-heading2"><h2 id="Literature">Literature</h2></div>

<ul><li>Multi-Target Prediction: A Unifying View on Problems and Methods Willem Waegeman, Krzysztof Dembczynski, Eyke Huellermeier <a rel="nofollow" class="external free" href="https://arxiv.org/abs/1809.02352v1">https://arxiv.org/abs/1809.02352v1</a></li></ul>

/* start https://en.wikipedia.org/ */

.mw-parser-output .div-col{margin-top:0.3em;column-width:30em}.mw-parser-output .div-col-small{font-size:90%}.mw-parser-output .div-col-rules{column-rule:1px solid #aaa}.mw-parser-output .div-col dl,.mw-parser-output .div-col ol,.mw-parser-output .div-col ul{margin-top:0}.mw-parser-output .div-col li,.mw-parser-output .div-col dd{page-break-inside:avoid;break-inside:avoid-column}

/* end https://en.wikipedia.org/ */

</style><div class="div-col">

<ul><li><a href="Artificial_intelligence" title="Artificial intelligence">Artificial intelligence</a></li>

<li><a href="Artificial_neural_network" class="mw-redirect" title="Artificial neural network">Artificial neural network</a></li>

<li><a href="Automated_machine_learning" title="Automated machine learning">Automated machine learning</a> (AutoML)</li>

<li><a href="Evolutionary_computation" title="Evolutionary computation">Evolutionary computation</a></li>

<li><a href="Foundation_model" title="Foundation model">Foundation model</a></li>

<li><a href="General_game_playing" title="General game playing">General game playing</a></li>

<li><a href="Human-based_genetic_algorithm" title="Human-based genetic algorithm">Human-based genetic algorithm</a></li>

<li><a href="Kernel_methods_for_vector_output" title="Kernel methods for vector output">Kernel methods for vector output</a></li>

<li><a href="Multiple-criteria_decision_analysis" title="Multiple-criteria decision analysis">Multiple-criteria decision analysis</a></li>

<li><a href="Multi-objective_optimization" title="Multi-objective optimization">Multi-objective optimization</a></li>

<li><a href="Multicriteria_classification" title="Multicriteria classification">Multicriteria classification</a></li>

<li><a href="Robot_learning" title="Robot learning">Robot learning</a></li>

<li><a href="Transfer_learning" title="Transfer learning">Transfer learning</a></li>

<li><a href="James%E2%80%93Stein_estimator" title="James–Stein estimator">James–Stein estimator</a></li></ul>

</div>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist">

<li id="cite_note-1"><span class="mw-cite-backlink"><b><a href="#cite_ref-1">^</a></b></span> <span class="reference-text">Baxter, J. (2000). A model of inductive bias learning" <i>Journal of Artificial Intelligence Research</i> 12:149--198, <a rel="nofollow" class="external text" href="http://www-2.cs.cmu.edu/afs/cs/project/jair/pub/volume12/baxter00a.pdf">On-line paper</a></span>

</li>

<li id="cite_note-2"><span class="mw-cite-backlink"><b><a href="#cite_ref-2">^</a></b></span> <span class="reference-text"><a href="Sebastian_Thrun" title="Sebastian Thrun">Thrun, S.</a> (1996). Is learning the n-th thing any easier than learning the first?. In Advances in Neural Information Processing Systems 8, pp. 640--646. MIT Press. <a rel="nofollow" class="external text" href="http://citeseer.ist.psu.edu/thrun96is.html">Paper at Citeseer</a></span>

</li>

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite id="CITEREFCaruana1997" class="citation journal cs1">Caruana, R. (1997). <a rel="nofollow" class="external text" href="http://www.cs.cornell.edu/~caruana/mlj97.pdf">"Multi-task learning"</a> <span class="cs1-format">(PDF)</span>. <i>Machine Learning</i>. <b>28</b>: <span class="nowrap">41–</span>75. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://doi.org/10.1023%2FA%3A1007379606734">10.1023/A:1007379606734</a></span>.</cite></span>

</li>

<li id="cite_note-4"><span class="mw-cite-backlink"><b><a href="#cite_ref-4">^</a></b></span> <span class="reference-text">Multi-Task Learning as Multi-Objective Optimization

Part of Advances in Neural Information Processing Systems 31 (NeurIPS 2018), <a rel="nofollow" class="external free" href="https://proceedings.neurips.cc/paper/2018/hash/432aca3a1e345e339f35a30c8f65edce-Abstract.html">https://proceedings.neurips.cc/paper/2018/hash/432aca3a1e345e339f35a30c8f65edce-Abstract.html</a></span>

</li>

<li id="cite_note-5"><span class="mw-cite-backlink"><b><a href="#cite_ref-5">^</a></b></span> <span class="reference-text">Suddarth, S., Kergosien, Y. (1990). Rule-injection hints as a means of improving network performance and learning time. EURASIP Workshop. Neural Networks pp. 120-129. Lecture Notes in Computer Science. Springer.</span>

</li>

<li id="cite_note-6"><span class="mw-cite-backlink"><b><a href="#cite_ref-6">^</a></b></span> <span class="reference-text"><cite id="CITEREFAbu-Mostafa1990" class="citation journal cs1">Abu-Mostafa, Y. S. (1990). <a rel="nofollow" class="external text" href="https://doi.org/10.1016%2F0885-064x%2890%2990006-y">"Learning from hints in neural networks"</a>. <i>Journal of Complexity</i>. <b>6</b> (2): <span class="nowrap">192–</span>198. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://doi.org/10.1016%2F0885-064x%2890%2990006-y">10.1016/0885-064x(90)90006-y</a></span>.</cite></span>

</li>

<li id="cite_note-:1-7"><span class="mw-cite-backlink">^ <a href="#cite_ref-:1_7-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-:1_7-1"><sup><i><b>b</b></i></sup></a> <a href="#cite_ref-:1_7-2"><sup><i><b>c</b></i></sup></a></span> <span class="reference-text"><cite id="CITEREFCiliberto2015" class="citation arxiv cs1">Ciliberto, C. (2015). "Convex Learning of Multiple Tasks and their Structure". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://arxiv.org/abs/1504.03101">1504.03101</a></span> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite></span>

</li>

<li id="cite_note-:bmdl-8"><span class="mw-cite-backlink">^ <a href="#cite_ref-:bmdl_8-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-:bmdl_8-1"><sup><i><b>b</b></i></sup></a> <a href="#cite_ref-:bmdl_8-2"><sup><i><b>c</b></i></sup></a> <a href="#cite_ref-:bmdl_8-3"><sup><i><b>d</b></i></sup></a></span> <span class="reference-text">Hajiramezanali, E. & Dadaneh, S. Z. & Karbalayghareh, A. & Zhou, Z. & Qian, X. Bayesian multi-domain learning for cancer subtype discovery from next-generation sequencing count data. 32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://arxiv.org/abs/1810.09433">1810.09433</a></span></span>

</li>

<li id="cite_note-:3-9"><span class="mw-cite-backlink">^ <a href="#cite_ref-:3_9-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-:3_9-1"><sup><i><b>b</b></i></sup></a></span> <span class="reference-text">Romera-Paredes, B., Argyriou, A., Bianchi-Berthouze, N., & Pontil, M., (2012) Exploiting Unrelated Tasks in Multi-Task Learning. <a rel="nofollow" class="external free" href="http://jmlr.csail.mit.edu/proceedings/papers/v22/romera12/romera12.pdf">http://jmlr.csail.mit.edu/proceedings/papers/v22/romera12/romera12.pdf</a></span>

</li>

<li id="cite_note-10"><span class="mw-cite-backlink"><b><a href="#cite_ref-10">^</a></b></span> <span class="reference-text">Kumar, A., & Daume III, H., (2012) Learning Task Grouping and Overlap in Multi-Task Learning. <a rel="nofollow" class="external free" href="http://icml.cc/2012/papers/690.pdf">http://icml.cc/2012/papers/690.pdf</a></span>

</li>

<li id="cite_note-11"><span class="mw-cite-backlink"><b><a href="#cite_ref-11">^</a></b></span> <span class="reference-text">Jawanpuria, P., & Saketha Nath, J., (2012) A Convex Feature Learning Formulation for Latent Task Structure Discovery. <a rel="nofollow" class="external free" href="http://icml.cc/2012/papers/90.pdf">http://icml.cc/2012/papers/90.pdf</a></span>

</li>

<li id="cite_note-12"><span class="mw-cite-backlink"><b><a href="#cite_ref-12">^</a></b></span> <span class="reference-text">Zweig, A. & Weinshall, D. Hierarchical Regularization Cascade for Joint Learning. Proceedings: of 30th International Conference on Machine Learning, Atlanta GA, June 2013. <a rel="nofollow" class="external free" href="http://www.cs.huji.ac.il/~daphna/papers/Zweig_ICML2013.pdf">http://www.cs.huji.ac.il/~daphna/papers/Zweig_ICML2013.pdf</a></span>

</li>

<li id="cite_note-13"><span class="mw-cite-backlink"><b><a href="#cite_ref-13">^</a></b></span> <span class="reference-text"><cite id="CITEREFSzegedyWei_LiuYangqing_JiaSermanet2015" class="citation book cs1">Szegedy, Christian; Wei Liu, Youssef; Yangqing Jia, Tomaso; Sermanet, Pierre; Reed, Scott; Anguelov, Dragomir; Erhan, Dumitru; Vanhoucke, Vincent; Rabinovich, Andrew (2015). "Going deeper with convolutions". <i>2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)</i>. pp. <span class="nowrap">1–</span>9. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://arxiv.org/abs/1409.4842">1409.4842</a></span>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FCVPR.2015.7298594">10.1109/CVPR.2015.7298594</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-1-4673-6964-0</bdi>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:206592484">206592484</a>.</cite></span>

</li>

<li id="cite_note-14"><span class="mw-cite-backlink"><b><a href="#cite_ref-14">^</a></b></span> <span class="reference-text"><cite id="CITEREFRoig" class="citation web cs1">Roig, Gemma. <a rel="nofollow" class="external text" href="https://web.archive.org/web/20160306020712/http://www.mit.edu/~9.520/fall15/slides/class24/deep_learning_overview.pdf">"Deep Learning Overview"</a> <span class="cs1-format">(PDF)</span>. Archived from <a rel="nofollow" class="external text" href="https://www.mit.edu/~9.520/fall15/slides/class24/deep_learning_overview.pdf">the original</a> <span class="cs1-format">(PDF)</span> on 2016-03-06<span class="reference-accessdate">. Retrieved <span class="nowrap">2019-08-26</span></span>.</cite></span>

</li>

<li id="cite_note-15"><span class="mw-cite-backlink"><b><a href="#cite_ref-15">^</a></b></span> <span class="reference-text">Zweig, A. & Chechik, G. Group online adaptive learning. Machine Learning, DOI 10.1007/s10994-017- 5661-5, August 2017. <a rel="nofollow" class="external free" href="http://rdcu.be/uFSv">http://rdcu.be/uFSv</a></span>

</li>

<li id="cite_note-TO-16"><span class="mw-cite-backlink">^ <a href="#cite_ref-TO_16-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-TO_16-1"><sup><i><b>b</b></i></sup></a></span> <span class="reference-text"><cite id="CITEREFGuptaOngFeng2018" class="citation journal cs1">Gupta, Abhishek; Ong, Yew-Soon; Feng, Liang (2018). "Insights on Transfer Optimization: Because Experience is the Best Teacher". <i>IEEE Transactions on Emerging Topics in Computational Intelligence</i>. <b>2</b>: <span class="nowrap">51–</span>64. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTETCI.2017.2769104">10.1109/TETCI.2017.2769104</a>. <a href="Hdl_(identifier)" class="mw-redirect" title="Hdl (identifier)">hdl</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://hdl.handle.net/10356%2F147980">10356/147980</a></span>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:11510470">11510470</a>.</cite></span>

</li>

<li id="cite_note-mfo-17"><span class="mw-cite-backlink">^ <a href="#cite_ref-mfo_17-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-mfo_17-1"><sup><i><b>b</b></i></sup></a> <a href="#cite_ref-mfo_17-2"><sup><i><b>c</b></i></sup></a></span> <span class="reference-text"><cite id="CITEREFGuptaOngFeng2016" class="citation journal cs1">Gupta, Abhishek; Ong, Yew-Soon; Feng, Liang (2016). "Multifactorial Evolution: Toward Evolutionary Multitasking". <i>IEEE Transactions on Evolutionary Computation</i>. <b>20</b> (3): <span class="nowrap">343–</span>357. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTEVC.2015.2458037">10.1109/TEVC.2015.2458037</a>. <a href="Hdl_(identifier)" class="mw-redirect" title="Hdl (identifier)">hdl</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://hdl.handle.net/10356%2F148174">10356/148174</a></span>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:13767012">13767012</a>.</cite></span>

</li>

<li id="cite_note-18"><span class="mw-cite-backlink"><b><a href="#cite_ref-18">^</a></b></span> <span class="reference-text"><cite id="CITEREFPanYang2010" class="citation journal cs1">Pan, Sinno Jialin; Yang, Qiang (2010). "A Survey on Transfer Learning". <i>IEEE Transactions on Knowledge and Data Engineering</i>. <b>22</b> (10): <span class="nowrap">1345–</span>1359. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTKDE.2009.191">10.1109/TKDE.2009.191</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:740063">740063</a>.</cite></span>

</li>

<li id="cite_note-19"><span class="mw-cite-backlink"><b><a href="#cite_ref-19">^</a></b></span> <span class="reference-text">Caruana, R., "Multitask Learning", pp. 95-134 in Sebastian Thrun, Lorien Pratt (eds.) <i>Learning to Learn</i>, (1998) Springer <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>9780792380474</bdi></span>

</li>

<li id="cite_note-20"><span class="mw-cite-backlink"><b><a href="#cite_ref-20">^</a></b></span> <span class="reference-text"><cite id="CITEREFChengGuptaOngNi2017" class="citation journal cs1">Cheng, Mei-Ying; Gupta, Abhishek; Ong, Yew-Soon; Ni, Zhi-Wei (2017). <a rel="nofollow" class="external text" href="https://doi.org/10.1016%2Fj.engappai.2017.05.008">"Coevolutionary multitasking for concurrent global optimization: With case studies in complex engineering design"</a>. <i>Engineering Applications of Artificial Intelligence</i>. <b>64</b>: <span class="nowrap">13–</span>24. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://doi.org/10.1016%2Fj.engappai.2017.05.008">10.1016/j.engappai.2017.05.008</a></span>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:13767210">13767210</a>.</cite></span>

</li>

<li id="cite_note-DeFreitas-21"><span class="mw-cite-backlink"><b><a href="#cite_ref-DeFreitas_21-0">^</a></b></span> <span class="reference-text"><cite id="CITEREFCabiSergio_Gómez_ColmenarejoHoffmanDenil2017" class="citation arxiv cs1">Cabi, Serkan; Sergio Gómez Colmenarejo; Hoffman, Matthew W.; Denil, Misha; Wang, Ziyu; Nando de Freitas (2017). "The Intentional Unintentional Agent: Learning to Solve Many Continuous Control Tasks Simultaneously". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://arxiv.org/abs/1707.03300">1707.03300</a></span> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.AI">cs.AI</a>].</cite></span>

</li>

<li id="cite_note-22"><span class="mw-cite-backlink"><b><a href="#cite_ref-22">^</a></b></span> <span class="reference-text">J. -Y. Li, Z. -H. Zhan, Y. Li and J. Zhang, "Multiple Tasks for Multiple Objectives: A New Multiobjective Optimization Method via Multitask Optimization," in IEEE Transactions on Evolutionary Computation, <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTEVC.2023.3294307">10.1109/TEVC.2023.3294307</a></span>

</li>

<li id="cite_note-23"><span class="mw-cite-backlink"><b><a href="#cite_ref-23">^</a></b></span> <span class="reference-text"><cite id="CITEREFStandleyZamirChenGuibas2020" class="citation journal cs1">Standley, Trevor; Zamir, Amir R.; Chen, Dawn; Guibas, Leonidas; Malik, Jitendra; Savarese, Silvio (2020-07-13). <a rel="nofollow" class="external text" href="https://proceedings.mlr.press/v119/standley20a.html">"Learning the Pareto Front with Hypernetworks"</a>. <i>International Conference on Machine Learning</i>: <span class="nowrap">9120–</span>9132. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://arxiv.org/abs/1905.07553">1905.07553</a></span>.</cite></span>

</li>

<li id="cite_note-mtbo-24"><span class="mw-cite-backlink"><b><a href="#cite_ref-mtbo_24-0">^</a></b></span> <span class="reference-text">Swersky, K., Snoek, J., & Adams, R. P. (2013). <a rel="nofollow" class="external text" href="http://papers.nips.cc/paper/5086-multi-task-bayesian-optimization.pdf">Multi-task bayesian optimization</a>. Advances in neural information processing systems (pp. 2004-2012).</span>

</li>

<li id="cite_note-25"><span class="mw-cite-backlink"><b><a href="#cite_ref-25">^</a></b></span> <span class="reference-text">Bonilla, E. V., Chai, K. M., & Williams, C. (2008). <a rel="nofollow" class="external text" href="http://papers.nips.cc/paper/3189-multi-task-gaussian-process-prediction.pdf">Multi-task Gaussian process prediction</a>. Advances in neural information processing systems (pp. 153-160).</span>

</li>

<li id="cite_note-cognitive-26"><span class="mw-cite-backlink">^ <a href="#cite_ref-cognitive_26-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-cognitive_26-1"><sup><i><b>b</b></i></sup></a></span> <span class="reference-text">Ong, Y. S., & Gupta, A. (2016). <a rel="nofollow" class="external text" href="http://www.cil.ntu.edu.sg/mfo/downloads/MultitaskOptimization_manuscript.pdf">Evolutionary multitasking: a computer science view of cognitive multitasking</a>. Cognitive Computation, 8(2), 125-142.</span>

</li>

<li id="cite_note-27"><span class="mw-cite-backlink"><b><a href="#cite_ref-27">^</a></b></span> <span class="reference-text"><cite id="CITEREFFengZhouZhongGupta2019" class="citation journal cs1">Feng, Liang; Zhou, Lei; Zhong, Jinghui; Gupta, Abhishek; Ong, Yew-Soon; Tan, Kay-Chen; Qin, A. K. (2019). "Evolutionary Multitasking via Explicit Autoencoding". <i>IEEE Transactions on Cybernetics</i>. <b>49</b> (9): <span class="nowrap">3457–</span>3470. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTCYB.2018.2845361">10.1109/TCYB.2018.2845361</a>. <a href="PMID_(identifier)" class="mw-redirect" title="PMID (identifier)">PMID</a> <a rel="nofollow" class="external text" href="https://pubmed.ncbi.nlm.nih.gov/29994415">29994415</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:51613697">51613697</a>.</cite></span>

</li>

<li id="cite_note-28"><span class="mw-cite-backlink"><b><a href="#cite_ref-28">^</a></b></span> <span class="reference-text"><cite id="CITEREFJiangZhanTanZhang2024" class="citation journal cs1">Jiang, Yi; Zhan, Zhi-Hui; Tan, Kay Chen; Zhang, Jun (January 2024). "Block-Level Knowledge Transfer for Evolutionary Multitask Optimization". <i>IEEE Transactions on Cybernetics</i>. <b>54</b> (1): <span class="nowrap">558–</span>571. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTCYB.2023.3273625">10.1109/TCYB.2023.3273625</a>. <a href="ISSN_(identifier)" class="mw-redirect" title="ISSN (identifier)">ISSN</a> <a rel="nofollow" class="external text" href="https://search.worldcat.org/issn/2168-2267">2168-2267</a>. <a href="PMID_(identifier)" class="mw-redirect" title="PMID (identifier)">PMID</a> <a rel="nofollow" class="external text" href="https://pubmed.ncbi.nlm.nih.gov/37216256">37216256</a>.</cite></span>

</li>

<li id="cite_note-29"><span class="mw-cite-backlink"><b><a href="#cite_ref-29">^</a></b></span> <span class="reference-text"><cite id="CITEREFGoodfellowBengioCourville2016" class="citation book cs1">Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016). <i>Deep Learning</i>. MIT Press. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-262-03561-3</bdi>.</cite></span>

</li>

<li id="cite_note-30"><span class="mw-cite-backlink"><b><a href="#cite_ref-30">^</a></b></span> <span class="reference-text"><cite id="CITEREFLiuLiKuangXue2021" class="citation web cs1">Liu, L.; Li, Y.; Kuang, Z.; Xue, J.; Chen, Y.; Yang, W.; Liao, Q.; Zhang, W. (2021-05-04). <a rel="nofollow" class="external text" href="https://iclr.cc/">"Towards Impartial Multi-task Learning"</a>. <i>In: Proceedings of the International Conference on Learning Representations (ICLR 2021). ICLR: Virtual event. (2021)</i><span class="reference-accessdate">. Retrieved <span class="nowrap">2022-11-20</span></span>.</cite></span>

</li>

<li id="cite_note-31"><span class="mw-cite-backlink"><b><a href="#cite_ref-31">^</a></b></span> <span class="reference-text"><cite id="CITEREFTianheSaurabhAbhishekSergey2020" class="citation journal cs1">Tianhe, Yu; Saurabh, Kumar; Abhishek, Gupta; Sergey, Levine; Karol, Hausman; Chelsea, Finn (2020). <a rel="nofollow" class="external text" href="https://proceedings.neurips.cc/paper/2020/hash/3fe78a8acf5fda99de95303940a2420c-Abstract.html">"Gradient Surgery for Multi-Task Learning"</a>. <i>Advances in Neural Information Processing Systems</i>. <b>33</b>. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://arxiv.org/abs/2001.06782">2001.06782</a></span>.</cite></span>

</li>

<li id="cite_note-32"><span class="mw-cite-backlink"><b><a href="#cite_ref-32">^</a></b></span> <span class="reference-text"><cite id="CITEREFLiuLiuJinStone2021" class="citation arxiv cs1">Liu, Bo; Liu, Xingchao; Jin, Xiaojie; Stone, Peter; Liu, Qiang (2021-10-26). "Conflict-Averse Gradient Descent for Multi-task Learning". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://arxiv.org/abs/2110.14048">2110.14048</a></span> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite></span>

</li>

<li id="cite_note-33"><span class="mw-cite-backlink"><b><a href="#cite_ref-33">^</a></b></span> <span class="reference-text">Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, Ethan Fetaya, (2022). <a rel="nofollow" class="external text" href="https://proceedings.mlr.press/v162/navon22a.html">Multi-Task Learning as a Bargaining Game</a>. International conference on machine learning.</span>

</li>

<li id="cite_note-34"><span class="mw-cite-backlink"><b><a href="#cite_ref-34">^</a></b></span> <span class="reference-text">Chandra, R., Gupta, A., Ong, Y. S., & Goh, C. K. (2016, October). <a rel="nofollow" class="external text" href="http://www.cil.ntu.edu.sg/mfo/downloads/cvmultask.pdf">Evolutionary multi-task learning for modular training of feedforward neural networks</a>. In International Conference on Neural Information Processing (pp. 37-46). Springer, Cham.</span>

</li>

<li id="cite_note-35"><span class="mw-cite-backlink"><b><a href="#cite_ref-35">^</a></b></span> <span class="reference-text">Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014). <a rel="nofollow" class="external text" href="http://papers.nips.cc/paper/5347-how-transferable-are-features-in-deep-n%E2%80%A6">How transferable are features in deep neural networks?</a> In Advances in neural information processing systems (pp. 3320-3328).</span>

</li>

<li id="cite_note-36"><span class="mw-cite-backlink"><b><a href="#cite_ref-36">^</a></b></span> <span class="reference-text"><cite id="CITEREFWenTing2016" class="citation book cs1">Wen, Yu-Wei; Ting, Chuan-Kang (2016). "Learning ensemble of decision trees through multifactorial genetic programming". <i>2016 IEEE Congress on Evolutionary Computation (CEC)</i>. pp. <span class="nowrap">5293–</span>5300. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FCEC.2016.7748363">10.1109/CEC.2016.7748363</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-1-5090-0623-6</bdi>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:2617811">2617811</a>.</cite></span>

</li>

<li id="cite_note-37"><span class="mw-cite-backlink"><b><a href="#cite_ref-37">^</a></b></span> <span class="reference-text"><cite id="CITEREFZhangQinSellis2018" class="citation book cs1">Zhang, Boyu; Qin, A. K.; Sellis, Timos (2018). "Evolutionary feature subspaces generation for ensemble classification". <i>Proceedings of the Genetic and Evolutionary Computation Conference</i>. pp. <span class="nowrap">577–</span>584. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1145%2F3205455.3205638">10.1145/3205455.3205638</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-1-4503-5618-3</bdi>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:49564862">49564862</a>.</cite></span>

</li>

<li id="cite_note-38"><span class="mw-cite-backlink"><b><a href="#cite_ref-38">^</a></b></span> <span class="reference-text"><cite id="CITEREFBaoQiShenBu2018" class="citation book cs1">Bao, Liang; Qi, Yutao; Shen, Mengqing; Bu, Xiaoxuan; Yu, Jusheng; Li, Qian; Chen, Ping (2018). "An Evolutionary Multitasking Algorithm for Cloud Computing Service Composition". <i>Services – SERVICES 2018</i>. Lecture Notes in Computer Science. Vol. 10975. pp. <span class="nowrap">130–</span>144. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1007%2F978-3-319-94472-2_10">10.1007/978-3-319-94472-2_10</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-3-319-94471-5</bdi>.</cite></span>

</li>

<li id="cite_note-39"><span class="mw-cite-backlink"><b><a href="#cite_ref-39">^</a></b></span> <span class="reference-text">Tang, J., Chen, Y., Deng, Z., Xiang, Y., & Joy, C. P. (2018). <a rel="nofollow" class="external text" href="https://www.ijcai.org/proceedings/2018/0538.pdf">A Group-based Approach to Improve Multifactorial Evolutionary Algorithm</a>. In IJCAI (pp. 3870-3876).</span>

</li>

<li id="cite_note-40"><span class="mw-cite-backlink"><b><a href="#cite_ref-40">^</a></b></span> <span class="reference-text"><cite id="CITEREFFeltonWighLapkin2021" class="citation cs1">Felton, Kobi; Wigh, Daniel; Lapkin, Alexei (2021). "Multi-task Bayesian Optimization of Chemical Reactions". <i>chemRxiv</i>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://doi.org/10.26434%2Fchemrxiv.13250216.v2">10.26434/chemrxiv.13250216.v2</a></span>.</cite></span>

</li>

<li id="cite_note-41"><span class="mw-cite-backlink"><b><a href="#cite_ref-41">^</a></b></span> <span class="reference-text"><cite id="CITEREFJiangZhanTanZhang2023" class="citation journal cs1">Jiang, Yi; Zhan, Zhi-Hui; Tan, Kay Chen; Zhang, Jun (October 2023). <a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTEVC.2022.3210783">"A Bi-Objective Knowledge Transfer Framework for Evolutionary Many-Task Optimization"</a>. <i>IEEE Transactions on Evolutionary Computation</i>. <b>27</b> (5): <span class="nowrap">1514–</span>1528. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTEVC.2022.3210783">10.1109/TEVC.2022.3210783</a></span>. <a href="ISSN_(identifier)" class="mw-redirect" title="ISSN (identifier)">ISSN</a> <a rel="nofollow" class="external text" href="https://search.worldcat.org/issn/1089-778X">1089-778X</a>.</cite></span>

</li>

<li id="cite_note-42"><span class="mw-cite-backlink"><b><a href="#cite_ref-42">^</a></b></span> <span class="reference-text"><cite id="CITEREFJiangZhanTanKwong2024" class="citation journal cs1">Jiang, Yi; Zhan, Zhi-Hui; Tan, Kay Chen; Kwong, Sam; Zhang, Jun (2024). <a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTEVC.2024.3355781">"Knowledge Structure Preserving-Based Evolutionary Many-Task Optimization"</a>. <i>IEEE Transactions on Evolutionary Computation</i>. <b>29</b> (2): <span class="nowrap">287–</span>301. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTEVC.2024.3355781">10.1109/TEVC.2024.3355781</a></span>. <a href="ISSN_(identifier)" class="mw-redirect" title="ISSN (identifier)">ISSN</a> <a rel="nofollow" class="external text" href="https://search.worldcat.org/issn/1089-778X">1089-778X</a>.</cite></span>

</li>

<li id="cite_note-43"><span class="mw-cite-backlink"><b><a href="#cite_ref-43">^</a></b></span> <span class="reference-text"><cite id="CITEREFDinuzzo2011" class="citation journal cs1">Dinuzzo, Francesco (2011). <a rel="nofollow" class="external text" href="https://web.archive.org/web/20170808223410/http://machinelearning.wustl.edu/mlpapers/paper_files/ICML2011Dinuzzo_54.pdf">"Learning output kernels with block coordinate descent"</a> <span class="cs1-format">(PDF)</span>. <i>Proceedings of the 28th International Conference on Machine Learning (ICML-11)</i>. Archived from <a rel="nofollow" class="external text" href="http://machinelearning.wustl.edu/mlpapers/paper_files/ICML2011Dinuzzo_54.pdf">the original</a> <span class="cs1-format">(PDF)</span> on 2017-08-08.</cite></span>

</li>

<li id="cite_note-44"><span class="mw-cite-backlink"><b><a href="#cite_ref-44">^</a></b></span> <span class="reference-text"><cite id="CITEREFJacob2009" class="citation journal cs1">Jacob, Laurent (2009). "Clustered multi-task learning: A convex formulation". <i>Advances in Neural Information Processing Systems</i>. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://arxiv.org/abs/0809.2085">0809.2085</a></span>. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2008arXiv0809.2085J">2008arXiv0809.2085J</a>.</cite></span>

</li>

<li id="cite_note-45"><span class="mw-cite-backlink"><b><a href="#cite_ref-45">^</a></b></span> <span class="reference-text">Zhou, J., Chen, J. and Ye, J. MALSAR: Multi-tAsk Learning via StructurAl Regularization. Arizona State University, 2012. <a rel="nofollow" class="external free" href="http://www.public.asu.edu/~jye02/Software/MALSAR">http://www.public.asu.edu/~jye02/Software/MALSAR</a>. <a rel="nofollow" class="external text" href="http://www.public.asu.edu/~jye02/Software/MALSAR/Manual.pdf">On-line manual</a></span>

</li>

<li id="cite_note-46"><span class="mw-cite-backlink"><b><a href="#cite_ref-46">^</a></b></span> <span class="reference-text">Evgeniou, T., & Pontil, M. (2004). <a rel="nofollow" class="external text" href="https://web.archive.org/web/20171212193041/https://pdfs.semanticscholar.org/1ea1/91c70559d21be93a4d128f95943e80e1b4ff.pdf">Regularized multi–task learning</a>. Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 109–117).</span>

</li>

<li id="cite_note-47"><span class="mw-cite-backlink"><b><a href="#cite_ref-47">^</a></b></span> <span class="reference-text"><cite id="CITEREFEvgeniouMicchelliPontil2005" class="citation journal cs1">Evgeniou, T.; Micchelli, C.; Pontil, M. (2005). <a rel="nofollow" class="external text" href="http://jmlr.org/papers/volume6/evgeniou05a/evgeniou05a.pdf">"Learning multiple tasks with kernel methods"</a> <span class="cs1-format">(PDF)</span>. <i>Journal of Machine Learning Research</i>. <b>6</b>: 615.</cite></span>

</li>

<li id="cite_note-48"><span class="mw-cite-backlink"><b><a href="#cite_ref-48">^</a></b></span> <span class="reference-text"><cite id="CITEREFArgyriouEvgeniouPontil2008a" class="citation journal cs1">Argyriou, A.; Evgeniou, T.; Pontil, M. (2008a). <a rel="nofollow" class="external text" href="https://doi.org/10.1007%2Fs10994-007-5040-8">"Convex multi-task feature learning"</a>. <i>Machine Learning</i>. <b>73</b> (3): <span class="nowrap">243–</span>272. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://doi.org/10.1007%2Fs10994-007-5040-8">10.1007/s10994-007-5040-8</a></span>.</cite></span>

</li>

<li id="cite_note-49"><span class="mw-cite-backlink"><b><a href="#cite_ref-49">^</a></b></span> <span class="reference-text">Chen, J., Zhou, J., & Ye, J. (2011). <a rel="nofollow" class="external text" href="https://www.academia.edu/download/44101186/Integrating_low-rank_and_group-sparse_st20160325-15067-1mftmbg.pdf">Integrating low-rank and group-sparse structures for robust multi-task learning</a>. Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining.</span>

</li>

<li id="cite_note-50"><span class="mw-cite-backlink"><b><a href="#cite_ref-50">^</a></b></span> <span class="reference-text">Ji, S., & Ye, J. (2009). <a rel="nofollow" class="external text" href="http://www.machinelearning.org/archive/icml2009/papers/151.pdf">An accelerated gradient method for trace norm minimization</a>. Proceedings of the 26th Annual International Conference on Machine Learning (pp. 457–464).</span>

</li>

<li id="cite_note-51"><span class="mw-cite-backlink"><b><a href="#cite_ref-51">^</a></b></span> <span class="reference-text"><cite id="CITEREFAndoZhang2005" class="citation journal cs1">Ando, R.; Zhang, T. (2005). <a rel="nofollow" class="external text" href="http://www.jmlr.org/papers/volume6/ando05a/ando05a.pdf">"A framework for learning predictive structures from multiple tasks and unlabeled data"</a> <span class="cs1-format">(PDF)</span>. <i>The Journal of Machine Learning Research</i>. <b>6</b>: <span class="nowrap">1817–</span>1853.</cite></span>

</li>

<li id="cite_note-52"><span class="mw-cite-backlink"><b><a href="#cite_ref-52">^</a></b></span> <span class="reference-text">Chen, J., Tang, L., Liu, J., & Ye, J. (2009). <a rel="nofollow" class="external text" href="http://leitang.net/papers/ICML09_CASO.pdf">A convex formulation for learning shared structures from multiple tasks</a>. Proceedings of the 26th Annual International Conference on Machine Learning (pp. 137–144).</span>

</li>

<li id="cite_note-53"><span class="mw-cite-backlink"><b><a href="#cite_ref-53">^</a></b></span> <span class="reference-text">Chen, J., Liu, J., & Ye, J. (2010). <a rel="nofollow" class="external text" href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3783291/">Learning incoherent sparse and low-rank patterns from multiple tasks</a>. Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 1179–1188).</span>

</li>

<li id="cite_note-54"><span class="mw-cite-backlink"><b><a href="#cite_ref-54">^</a></b></span> <span class="reference-text">Jacob, L., Bach, F., & Vert, J. (2008). <a rel="nofollow" class="external text" href="https://hal-ensmp.archives-ouvertes.fr/docs/00/32/05/73/PDF/cmultitask.pdf">Clustered multi-task learning: A convex formulation</a>. Advances in Neural Information Processing Systems， 2008</span>

</li>

<li id="cite_note-55"><span class="mw-cite-backlink"><b><a href="#cite_ref-55">^</a></b></span> <span class="reference-text">Zhou, J., Chen, J., & Ye, J. (2011). <a rel="nofollow" class="external text" href="http://papers.nips.cc/paper/4292-clustered-multi-task-learning-via-alternating-structure-optimization.pdf">Clustered multi-task learning via alternating structure optimization</a>. Advances in Neural Information Processing Systems.</span>

</li>

</ol></div></div>

<div class="mw-heading mw-heading2"><h2 id="External_links">External links</h2></div>

<ul><li><a rel="nofollow" class="external text" href="https://web.archive.org/web/20041118134329/http://big.cs.uiuc.edu/webpage/cumulativeLearning/cumulativeLearning.html">The Biosignals Intelligence Group at UIUC</a></li>

<li><a rel="nofollow" class="external text" href="http://www.cse.wustl.edu/~kilian/research/multitasklearning/multitasklearning.html">Washington University in St. Louis Department of Computer Science</a></li></ul>

<div class="mw-heading mw-heading3"><h3 id="Software">Software</h3></div>

<ul><li><a rel="nofollow" class="external text" href="http://www.public.asu.edu/~jye02/Software/MALSAR/index.html">The Multi-Task Learning via Structural Regularization Package</a></li>

<li><a rel="nofollow" class="external text" href="https://web.archive.org/web/20131224113826/http://klcl.pku.edu.cn/member/sunxu/code.htm">Online Multi-Task Learning Toolkit (OMT)</a> A general-purpose online multi-task learning toolkit based on <a href="Conditional_random_field" title="Conditional random field">conditional random field</a> models and <a href="Stochastic_gradient_descent" title="Stochastic gradient descent">stochastic gradient descent</a> training (<a href="C_Sharp_(programming_language)" title="C Sharp (programming language)">C#</a>, <a href=".NET_Framework" title=".NET Framework">.NET</a>)</li></ul>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hlist dl,.mw-parser-output .hlist ol,.mw-parser-output .hlist ul{margin:0;padding:0}.mw-parser-output .hlist dd,.mw-parser-output .hlist dt,.mw-parser-output .hlist li{margin:0;display:inline}.mw-parser-output .hlist.inline,.mw-parser-output .hlist.inline dl,.mw-parser-output .hlist.inline ol,.mw-parser-output .hlist.inline ul,.mw-parser-output .hlist dl dl,.mw-parser-output .hlist dl ol,.mw-parser-output .hlist dl ul,.mw-parser-output .hlist ol dl,.mw-parser-output .hlist ol ol,.mw-parser-output .hlist ol ul,.mw-parser-output .hlist ul dl,.mw-parser-output .hlist ul ol,.mw-parser-output .hlist ul ul{display:inline}.mw-parser-output .hlist .mw-empty-li{display:none}.mw-parser-output .hlist dt::after{content:": "}.mw-parser-output .hlist dd::after,.mw-parser-output .hlist li::after{content:" · ";font-weight:bold}.mw-parser-output .hlist dd:last-child::after,.mw-parser-output .hlist dt:last-child::after,.mw-parser-output .hlist li:last-child::after{content:none}.mw-parser-output .hlist dd dd:first-child::before,.mw-parser-output .hlist dd dt:first-child::before,.mw-parser-output .hlist dd li:first-child::before,.mw-parser-output .hlist dt dd:first-child::before,.mw-parser-output .hlist dt dt:first-child::before,.mw-parser-output .hlist dt li:first-child::before,.mw-parser-output .hlist li dd:first-child::before,.mw-parser-output .hlist li dt:first-child::before,.mw-parser-output .hlist li li:first-child::before{content:" (";font-weight:normal}.mw-parser-output .hlist dd dd:last-child::after,.mw-parser-output .hlist dd dt:last-child::after,.mw-parser-output .hlist dd li:last-child::after,.mw-parser-output .hlist dt dd:last-child::after,.mw-parser-output .hlist dt dt:last-child::after,.mw-parser-output .hlist dt li:last-child::after,.mw-parser-output .hlist li dd:last-child::after,.mw-parser-output .hlist li dt:last-child::after,.mw-parser-output .hlist li li:last-child::after{content:")";font-weight:normal}.mw-parser-output .hlist ol{counter-reset:listitem}.mw-parser-output .hlist ol>li{counter-increment:listitem}.mw-parser-output .hlist ol>li::before{content:" "counter(listitem)"\a0 "}.mw-parser-output .hlist dd ol>li:first-child::before,.mw-parser-output .hlist dt ol>li:first-child::before,.mw-parser-output .hlist li ol>li:first-child::before{content:" ("counter(listitem)"\a0 "}

/* end https://en.wikipedia.org/ */

</style><style data-mw-deduplicate="TemplateStyles:r1236075235">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbox{box-sizing:border-box;border:1px solid #a2a9b1;width:100%;clear:both;font-size:88%;text-align:center;padding:1px;margin:1em auto 0}.mw-parser-output .navbox .navbox{margin-top:0}.mw-parser-output .navbox+.navbox,.mw-parser-output .navbox+.navbox-styles+.navbox{margin-top:-1px}.mw-parser-output .navbox-inner,.mw-parser-output .navbox-subgroup{width:100%}.mw-parser-output .navbox-group,.mw-parser-output .navbox-title,.mw-parser-output .navbox-abovebelow{padding:0.25em 1em;line-height:1.5em;text-align:center}.mw-parser-output .navbox-group{white-space:nowrap;text-align:right}.mw-parser-output .navbox,.mw-parser-output .navbox-subgroup{background-color:#fdfdfd}.mw-parser-output .navbox-list{line-height:1.5em;border-color:#fdfdfd}.mw-parser-output .navbox-list-with-group{text-align:left;border-left-width:2px;border-left-style:solid}.mw-parser-output tr+tr>.navbox-abovebelow,.mw-parser-output tr+tr>.navbox-group,.mw-parser-output tr+tr>.navbox-image,.mw-parser-output tr+tr>.navbox-list{border-top:2px solid #fdfdfd}.mw-parser-output .navbox-title{background-color:#ccf}.mw-parser-output .navbox-abovebelow,.mw-parser-output .navbox-group,.mw-parser-output .navbox-subgroup .navbox-title{background-color:#ddf}.mw-parser-output .navbox-subgroup .navbox-group,.mw-parser-output .navbox-subgroup .navbox-abovebelow{background-color:#e6e6ff}.mw-parser-output .navbox-even{background-color:#f7f7f7}.mw-parser-output .navbox-odd{background-color:transparent}.mw-parser-output .navbox .hlist td dl,.mw-parser-output .navbox .hlist td ol,.mw-parser-output .navbox .hlist td ul,.mw-parser-output .navbox td.hlist dl,.mw-parser-output .navbox td.hlist ol,.mw-parser-output .navbox td.hlist ul{padding:0.125em 0}.mw-parser-output .navbox .navbar{display:block;font-size:100%}.mw-parser-output .navbox-title .navbar{float:left;text-align:left;margin-right:0.5em}body.skin--responsive .mw-parser-output .navbox-image img{max-width:none!important}@media print{body.ns-0 .mw-parser-output .navbox{display:none!important}}

/* end https://en.wikipedia.org/ */

</style></div><div role="navigation" class="navbox" aria-labelledby="Optimization:_Algorithms,_methods,_and_heuristics381" style="padding:3px"><table class="nowraplinks hlist mw-collapsible uncollapsed navbox-inner" style="border-spacing:0;background:transparent;color:inherit"><tbody><tr><th scope="col" class="navbox-title" colspan="3"><style data-mw-deduplicate="TemplateStyles:r1239400231">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbar{display:inline;font-size:88%;font-weight:normal}.mw-parser-output .navbar-collapse{float:left;text-align:left}.mw-parser-output .navbar-boxtext{word-spacing:0}.mw-parser-output .navbar ul{display:inline-block;white-space:nowrap;line-height:inherit}.mw-parser-output .navbar-brackets::before{margin-right:-0.125em;content:"[ "}.mw-parser-output .navbar-brackets::after{margin-left:-0.125em;content:" ]"}.mw-parser-output .navbar li{word-spacing:-0.125em}.mw-parser-output .navbar a>span,.mw-parser-output .navbar a>abbr{text-decoration:inherit}.mw-parser-output .navbar-mini abbr{font-variant:small-caps;border-bottom:none;text-decoration:none;cursor:inherit}.mw-parser-output .navbar-ct-full{font-size:114%;margin:0 7em}.mw-parser-output .navbar-ct-mini{font-size:114%;margin:0 4em}html.skin-theme-clientpref-night .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}@media(prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}}@media print{.mw-parser-output .navbar{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div id="Optimization:_Algorithms,_methods,_and_heuristics381" style="font-size:114%;margin:0 4em"><a href="Mathematical_optimization" title="Mathematical optimization">Optimization</a>: <a href="Optimization_algorithm" class="mw-redirect" title="Optimization algorithm">Algorithms</a>, <a href="Iterative_method" title="Iterative method">methods</a>, and <a href="Heuristic_algorithm" class="mw-redirect" title="Heuristic algorithm">heuristics</a></div></th></tr><tr><td colspan="2" class="navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks mw-collapsible mw-collapsed navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="col" class="navbox-title" colspan="2"><div id="Unconstrained_nonlinear381" style="font-size:114%;margin:0 4em"><a href="Nonlinear_programming" title="Nonlinear programming">Unconstrained nonlinear</a></div></th></tr><tr><td colspan="2" class="navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Function_(mathematics)" title="Function (mathematics)">Functions</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Golden-section_search" title="Golden-section search">Golden-section search</a></li>

<li><a href="Powell's_method" title="Powell's method">Powell's method</a></li>

<li><a href="Line_search" title="Line search">Line search</a></li>

<li><a href="Nelder%E2%80%93Mead_method" title="Nelder–Mead method">Nelder–Mead method</a></li>

<li><a href="Successive_parabolic_interpolation" title="Successive parabolic interpolation">Successive parabolic interpolation</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Gradient" title="Gradient">Gradients</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Local_convergence" title="Local convergence">Convergence</a></th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Trust_region" title="Trust region">Trust region</a></li>

<li><a href="Wolfe_conditions" title="Wolfe conditions">Wolfe conditions</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Quasi-Newton_method" title="Quasi-Newton method">Quasi–Newton</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Berndt%E2%80%93Hall%E2%80%93Hall%E2%80%93Hausman_algorithm" title="Berndt–Hall–Hall–Hausman algorithm">Berndt–Hall–Hall–Hausman</a></li>

<li><a href="Broyden%E2%80%93Fletcher%E2%80%93Goldfarb%E2%80%93Shanno_algorithm" title="Broyden–Fletcher–Goldfarb–Shanno algorithm">Broyden–Fletcher–Goldfarb–Shanno</a> and <a href="Limited-memory_BFGS" title="Limited-memory BFGS">L-BFGS</a></li>

<li><a href="Davidon%E2%80%93Fletcher%E2%80%93Powell_formula" title="Davidon–Fletcher–Powell formula">Davidon–Fletcher–Powell</a></li>

<li><a href="Symmetric_rank-one" title="Symmetric rank-one">Symmetric rank-one (SR1)</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Iterative_method" title="Iterative method">Other methods</a></th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Nonlinear_conjugate_gradient_method" title="Nonlinear conjugate gradient method">Conjugate gradient</a></li>

<li><a href="Gauss%E2%80%93Newton_algorithm" title="Gauss–Newton algorithm">Gauss–Newton</a></li>

<li><a href="Gradient_descent" title="Gradient descent">Gradient</a></li>

<li><a href="Mirror_descent" title="Mirror descent">Mirror</a></li>

<li><a href="Levenberg%E2%80%93Marquardt_algorithm" title="Levenberg–Marquardt algorithm">Levenberg–Marquardt</a></li>

<li><a href="Powell's_dog_leg_method" title="Powell's dog leg method">Powell's dog leg method</a></li>

<li><a href="Truncated_Newton_method" title="Truncated Newton method">Truncated Newton</a></li></ul>

</div></td></tr></tbody></table><div></div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Hessian_matrix" title="Hessian matrix">Hessians</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Newton's_method_in_optimization" title="Newton's method in optimization">Newton's method</a></li></ul>

</div></td></tr></tbody></table><div></div></td></tr></tbody></table><div></div></td><td class="noviewer navbox-image" rowspan="5" style="width:1px;padding:0 0 0 2px"><div></div></td></tr><tr><td colspan="2" class="navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks mw-collapsible mw-collapsed navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="col" class="navbox-title" colspan="2"><div id="Constrained_nonlinear381" style="font-size:114%;margin:0 4em"><a href="Nonlinear_programming" title="Nonlinear programming">Constrained nonlinear</a></div></th></tr><tr><td colspan="2" class="navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="row" class="navbox-group" style="width:1%">General</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Barrier_function" title="Barrier function">Barrier methods</a></li>

<li><a href="Penalty_method" title="Penalty method">Penalty methods</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Differentiable</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Augmented_Lagrangian_method" title="Augmented Lagrangian method">Augmented Lagrangian methods</a></li>

<li><a href="Sequential_quadratic_programming" title="Sequential quadratic programming">Sequential quadratic programming</a></li>

<li><a href="Successive_linear_programming" title="Successive linear programming">Successive linear programming</a></li></ul>

</div></td></tr></tbody></table><div></div></td></tr></tbody></table><div></div></td></tr><tr><td colspan="2" class="navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks mw-collapsible mw-collapsed navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="col" class="navbox-title" colspan="2"><div id="Convex_optimization381" style="font-size:114%;margin:0 4em"><a href="Convex_optimization" title="Convex optimization">Convex optimization</a></div></th></tr><tr><td colspan="2" class="navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Convex_minimization" class="mw-redirect" title="Convex minimization">Convex<br> minimization</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Cutting-plane_method" title="Cutting-plane method">Cutting-plane method</a></li>

<li><a href="Frank%E2%80%93Wolfe_algorithm" title="Frank–Wolfe algorithm">Reduced gradient (Frank–Wolfe)</a></li>

<li><a href="Subgradient_method" title="Subgradient method">Subgradient method</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Linear_programming" title="Linear programming">Linear</a> and<br><a href="Quadratic_programming" title="Quadratic programming">quadratic</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Linear_programming#Interior_point" title="Linear programming">Interior point</a></th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Affine_scaling" title="Affine scaling">Affine scaling</a></li>

<li><a href="Ellipsoid_method" title="Ellipsoid method">Ellipsoid algorithm of Khachiyan</a></li>

<li><a href="Karmarkar's_algorithm" title="Karmarkar's algorithm">Projective algorithm of Karmarkar</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Matroid" title="Matroid">Basis-</a><a href="Exchange_algorithm" class="mw-redirect" title="Exchange algorithm">exchange</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Simplex_algorithm" title="Simplex algorithm">Simplex algorithm of Dantzig</a></li>

<li><a href="Revised_simplex_method" title="Revised simplex method">Revised simplex algorithm</a></li>

<li><a href="Criss-cross_algorithm" title="Criss-cross algorithm">Criss-cross algorithm</a></li>

<li><a href="Lemke's_algorithm" title="Lemke's algorithm">Principal pivoting algorithm of Lemke</a></li>

<li><a href="Active-set_method" title="Active-set method">Active-set method</a></li></ul>

</div></td></tr></tbody></table><div></div></td></tr></tbody></table><div></div></td></tr></tbody></table><div></div></td></tr><tr><td colspan="2" class="navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks mw-collapsible mw-collapsed navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="col" class="navbox-title" colspan="2"><div id="Combinatorial381" style="font-size:114%;margin:0 4em"><a href="Combinatorial_optimization" title="Combinatorial optimization">Combinatorial</a></div></th></tr><tr><td colspan="2" class="navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="row" class="navbox-group" style="width:1%">Paradigms</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Approximation_algorithm" title="Approximation algorithm">Approximation algorithm</a></li>

<li><a href="Dynamic_programming" title="Dynamic programming">Dynamic programming</a></li>

<li><a href="Greedy_algorithm" title="Greedy algorithm">Greedy algorithm</a></li>

<li><a href="Integer_programming" title="Integer programming">Integer programming</a>

<ul><li><a href="Branch_and_bound" title="Branch and bound">Branch and bound</a>/<a href="Branch_and_cut" title="Branch and cut">cut</a></li></ul></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Graph_algorithm" class="mw-redirect" title="Graph algorithm">Graph<br> algorithms</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th id="Minimum_spanning_tree52" scope="row" class="navbox-group" style="width:1%"><a href="Minimum_spanning_tree" title="Minimum spanning tree">Minimum<br> spanning tree</a></th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Bor%C5%AFvka's_algorithm" title="Borůvka's algorithm">Borůvka</a></li>

<li><a href="Kruskal's_algorithm" title="Kruskal's algorithm">Kruskal</a></li></ul>

</div></td></tr></tbody></table><div>

</div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th id="Shortest_path39" scope="row" class="navbox-group" style="width:1%"><a href="Shortest_path_problem" title="Shortest path problem">Shortest path</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Bellman%E2%80%93Ford_algorithm" title="Bellman–Ford algorithm">Bellman–Ford</a>

<li><a href="Dijkstra's_algorithm" title="Dijkstra's algorithm">Dijkstra</a></li>

<li><a href="Floyd%E2%80%93Warshall_algorithm" title="Floyd–Warshall algorithm">Floyd–Warshall</a></li></ul>

</div></td></tr></tbody></table><div></div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Flow_network" title="Flow network">Network flows</a></th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Dinic's_algorithm" title="Dinic's algorithm">Dinic</a></li>

<li><a href="Edmonds%E2%80%93Karp_algorithm" title="Edmonds–Karp algorithm">Edmonds–Karp</a></li>

<li><a href="Ford%E2%80%93Fulkerson_algorithm" title="Ford–Fulkerson algorithm">Ford–Fulkerson</a></li>

<li><a href="Push%E2%80%93relabel_maximum_flow_algorithm" title="Push–relabel maximum flow algorithm">Push–relabel maximum flow</a></li></ul>

</div></td></tr></tbody></table><div></div></td></tr></tbody></table><div></div></td></tr><tr><td colspan="2" class="navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks mw-collapsible mw-collapsed navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="col" class="navbox-title" colspan="2"><div id="Metaheuristics381" style="font-size:114%;margin:0 4em"><a href="Metaheuristic" title="Metaheuristic">Metaheuristics</a></div></th></tr><tr><td colspan="2" class="navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Evolutionary_algorithm" title="Evolutionary algorithm">Evolutionary algorithm</a></li>

<li><a href="Hill_climbing" title="Hill climbing">Hill climbing</a></li>

<li><a href="Local_search_(optimization)" title="Local search (optimization)">Local search</a></li>

<li><a href="Parallel_metaheuristic" title="Parallel metaheuristic">Parallel metaheuristics</a></li>

<li><a href="Simulated_annealing" title="Simulated annealing">Simulated annealing</a></li>

<li><a href="Spiral_optimization_algorithm" title="Spiral optimization algorithm">Spiral optimization algorithm</a></li>

<li><a href="Tabu_search" title="Tabu search">Tabu search</a></li></ul>

</div></td></tr></tbody></table><div></div></td></tr><tr><td class="navbox-abovebelow" colspan="3"><div>

<ul><li><a href="Comparison_of_optimization_software" title="Comparison of optimization software">Software</a></li></ul>

</div></td></tr></tbody></table></div></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2025-07-10" href="https://en.wikipedia.org/wiki/?title=Multi-task_learning&oldid=1299855426">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>